Best way to deploy an LLM for 100s of user

Question

If you have any experience of deploying an LLM what resources/tools did you use?do you have a link to your project?Anything will help.

eschnou · Accepted Answer

Could you detail what you mean by deploying LLMs ? Is it about integrating commercial LLMs in an enterprise context? Or running self-hosted LLM for a small company (e.g. Ollama + Ollama Web UI)? Or integrating an Agentic approach to existing software stack?

gardnr · Answer

Not enough info.Do they want near-realtime responses? Will they all hit it at the same time? Can you put some workloads in an overnight batch queue?

K0IN · Answer

so for fast responses we usea a rtx4090 with vllm, but yeah it depends on your use case

Best way to deploy an LLM for 100s of user

If you have any experience of deploying an LLM what resources/tools did you use?
do you have a link to your project?
Anything will help.

Could you detail what you mean by deploying LLMs ? Is it about integrating commercial LLMs in an enterprise context? Or running self-hosted LLM for a small company (e.g. Ollama + Ollama Web UI)? Or integrating an Agentic approach to existing software stack?

Not enough info.
Do they want near-realtime responses? Will they all hit it at the same time? Can you put some workloads in an overnight batch queue?

so for fast responses we usea a rtx4090 with vllm, but yeah it depends on your use case