If I were to build an LLM app today...

Question

There are so many libraries and frameworks out there, from LangChain to Ollama. While I could just use OpenAI's Assistants API, I want to compose my LLM responses in a way that's unique to my app; I don't want to build yet another thin wrapper around ChatGPT, where the only selling point is the proprietary prompt behind the app. Additionally, I would like to be able to switch to a different LLM provider (e.g. in case OpenAI fails on us or local LLMs have finally caught up to the same quality).What are your recommendations for AI libraries? Tell me about your experience.

bbsz · Accepted Answer

There's no reason to think that you can't craft unique "prompt flow" without any of mentioned libraries. Standard OpenAI library, "requests" and a long thought about what exactly do you need from prompt engineering in your app it's a time better spent than learning high-level representations and mundane of pretty big libraries. Remember that those will also obscure a lot useful knowledge for you in process and prompt engineering today is not that big of a field to benefit from such (I guess I wouldn't want to build neural networks without helper libraries)My experience with building around LLMs is that before you reach the prompt engineering phase and begin to craft fancy chain of reasoning flows on your data (if you won't discover a long the way that there are easier ways than LLM), you'll be pulling hairs for days trying to first define what type of a data and in what format to deliver to prompt so your output even falls within your base expectation.

meiraleal · Answer

The open AI API is becoming a default API. It's easier to create wrappers for each LLM than using langchain in my opinion, it's an unnecessary abstraction. The software around those wrappers is what matters, using langchain wouldn't change that

lgrammel · Answer

In case you're using JS/TS, I've been working on an abstraction layer for integrating AI models applications, unifying the API for common operations such as text streaming, object generation, and tool usage: https://github.com/lgrammel/modelfusion

altdataseller · Answer

Use the tool that best solves your problem. What&rsquo;s your problem? I would focus on that first