What's GPT-4's secret sauce that makes it so much better than others?

Question

Especially for coding, I've had absolutely fantastic results with GPT-4. I've tried ChatGPT, Claude, LlaMA, and others, but GPT-4 is the only LLM that performs superb in terms of coding. Most times I get the correct response at the first try.There's a rumor about the number of GPT-4's parameters (1 trillion?). But I'm not sure. To me, it's just an amazing piece of technology that works very well and somehow I don't know how it achieved this performance.

tikkun · Accepted Answer

OpenAI's mentioned that it was basically getting a lot of little details right.
Some of the bigger details:
* Huge compute scale for the pre-training
* Huge data scale
* Well processed and cleaned data (GDB has alluded to this)
* Fine tuning with large amounts of code snippets and examples
* Massive amounts of RLHF
* Continual ongoing RLHF