Browser-Based LLM Models?

Question

Does anyone know if there are there any plans for browsers to natively integrate LLMs, LLM APIs, or LLM models like Llama for local use by web applications?I feel there's a large opportunity here for a more privacy-friendly, on-device solution that doesn't send the user's data to OpenAI.Is RAM the current main limitation?

throwaway888abc · Accepted Answer

https://simonwillison.net/2024/Jul/3/chrome-prompt-playgroun...https://developer.chrome.com/docs/ai/built-in

throwaway425933 · Answer

Every big tech company is trying to do this. FB (through whatsapp), Google (through chrome/Android), Apple (through Safari/iOS/etc). As soon as they meet their internal metrics, they will release these to public

FrenchDevRemote · Answer

"Is RAM the current main limitation?"(V)RAM+processing power+storage(I mean what kind of average user wants to clog half their hard drive for a subpar model that output 1 token a second?)

Crier1002 · Answer

check out https://github.com/mlc-ai/web-llmIMO the main limitation is access to powerful GPUs for running models locally and the size of some models causing UX problems with cold starts

Browser-Based LLM Models?

https://simonwillison.net/2024/Jul/3/chrome-prompt-playgroun...
https://developer.chrome.com/docs/ai/built-in

Every big tech company is trying to do this. FB (through whatsapp), Google (through chrome/Android), Apple (through Safari/iOS/etc). As soon as they meet their internal metrics, they will release these to public

"Is RAM the current main limitation?"
(V)RAM+processing power+storage(I mean what kind of average user wants to clog half their hard drive for a subpar model that output 1 token a second?)

check out https://github.com/mlc-ai/web-llm
IMO the main limitation is access to powerful GPUs for running models locally and the size of some models causing UX problems with cold starts