What do you use local LLMs for?

- Basic internet search (I start ollama CLI faster than I can start a browser - https://ollama.com)

- Formatting/changing text

- Troubleshooting code, esp. new frameworks/libs

- Recipes

- Data entry

- Organizing thoughts: High-level lists, comparison, classification, synonyms, jargon & nomenclature

- Learning esp. by analogy and example

RAG for:

- Website assistants (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Game NPCs (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Discord/Slack/forum bots (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Character-driven storytelling and video game concept art (https://github.com/bennyschmidt/ragdoll-studio/tree/master/r...)

I use it as a daily driver (built https://recurse.chat/).

Local RAG and chat with PDF is handy. Some of our users are using it to format transcripts (example: https://talk.macpowerusers.com/t/recursechat-little-app-to-u...).

I use the MLCChat iOS app to run Mixtral on my iPhone pretty much exclusively when on flights and am curious about something. Maybe it’s a question about my destination, a fact I’ve forgotten, or if I’m working and haven’t bought WiFi it’s good for quick documentation sythesis.

It’s genuinely useful and fun in a way ChatGPT isn’t- something about the novelty. However outside of that environment I haven’t found a good use.

Also fun to feel the phone heat up as it generates tokens.

Edit: 7B-Instruct

The thought of building an interview assistant for coding interviews and selling it for few $k A pop has crossed my mind on more than one occasion. Run audio capture and OCR, along with some finetuning on prompt engineering of how to solve the problem in steps with explanation of each one.

I'm looking to implement a local RAG for my business to read, pdfs, spreadsheets, FAQ, and scan parent company websites.

Not sure where to start but people projects here are giving me a good direction.

Eventually I'd want the chat on my business site to start a sale process.

Using llm for converting LinkedIn post to Google doc

SBERT for classification, ranking, etc.