How to Fine Tune LLMs?

Question

I wanted to play around with fine tuning LLMs on my Mac. Any recommendations on how (or) where to start? Is there a blog you'd suggest?

PaulHoule · Accepted Answer

These instructions got me startedhttps://huggingface.co/docs/transformers/trainingwith the caveat that the default training rate is too high and you get better results with 2e-5

verdverm · Answer

There are a number of ways, with increasing methods and capabilities. It's going to depend on what you are trying to do and the constraints you haveGoogle put out a paper last week that talks about some of this and proposed a new method that outperforms previous: https://news.ycombinator.com/item?id=35810663