2. In general, what is the most "batteries included" way to work with agents? My goal is to reduce time spent integrating tools (logging, monitoring, evaluation, security, etc) while I build PoCs. My current approach is LiteLLM + LangFuse, but would love to head about something better.
After working with thousands of teams building agents and complex LLM applications, I'd conclude that:
- all frameworks have their strengths and weaknesses, Crew and LangGraph are the popular options if you want to use a framework, both have sdk methods or add-on packages to visualize the agent
- biggest overall strength is to get started quickly
- biggest overall weakness is that complexity is abstracted which can lead to building more complexity than necessary, making it harder to maintain
- many teams build light abstractions and do very well with them
- larger organizations benefit from frameworks as it standardizes how llm applications are built, thus being able to hire and move people between teams while relying on the same abstraction -- without needing to design a generalized abstraction that works for all sorts of use cases
- many use cases do not need an "agent", implementation can be simpler and more predictable
I think you'd enjoy this very recent podcast episode on one way of thinking about building agents with SOTA models: https://www.latent.space/p/claude-sonnet
tldl: many use cases need less abstraction than initially assumed when using SOTA models, focus on tools as they have a lot of leverage on performance/quality
PS: Would love to chat in case you are interested as we currently plan many improvements with regards to agents. I just saw your message on this in the acquired slack and responded there right before getting the HN notification.