How can I tell which LLM's have been trained on what?

Question

I'm really no expert and attempting to come up to speed. But...It strikes me that LLM's are a subtle combination of knowing the language and knowing some facts. If one were making a chatbot for helping (say) people understand their tax returns then you'd want a 'base' LLM that understood language and then add domain specific knowledge via RAG. So "yes" to knowing that Thursday comes after Wednesday; "no" to knowing how high Everest is; and questions regarding income tax have to reference the RAG. Is this correct?So are there stock "thin" models for doing this? Is it better to train your own?

uberman · Accepted Answer

I'm not exactly sure what you are asking. You can however construct a RAG based on a training corpus and then if you ask questions that when vecroized have no sufficiently similar match in your corpus you tell your llm to just say it cant answer that question.Concretely, index the tax documents into your data base then ask a question about Mount Everest. Vectorize that question and search your vectoized corpus for something similar. Find nothing then instruct your llm to formulate a response saying it could not answer

swrrt · Answer

Don't think RAG is the most common method for now. In my knowledge now it is mostly about post training and reenforcement learning for adding specific knowledge to base model. RAG is mostly like an add-on to a fully trained model and not so stable or inherent?

wmf · Answer

Yeah, the dream is to have a "reasoning kernel" that you can plug specific knowledge and skills into ("I know kung fu"). I'm not aware of any progress on the dream though. Plenty of people are making LoRAs for general-purpose open models and there are companies like Thinking Machines that will help you. I'm not sure how you'd choose which base model to start with. I'm also not sure how to draw the boundary between LoRA and RAG.
Some links I found but I can't vouch for them:
https://github.com/oobabooga/textgen/wiki/05-%E2%80%90-Train...
https://thinkingmachines.ai/tinker/