Is it hopeless to try to build new foundational models now

Question

Sama says in an answer to a question(What should Indian startups do to build foundational models?) that it is hopeless to try to build foundational models that can compete with OpenAI. What does the community think? Is OpenAI too far ahead? Is there a possibility of some new tech coming which can displace OpenAI?We are betting on the fact that such new tech is possible. Just wanted to get the feel of the community in this regard.

raxxorraxor · Accepted Answer

I think it is very much possible, especially in economies like India and China. If some business cases for Chat GPT are realized, I expect competitors to surface.
The a difficult part is probably getting high quality training data. And there may very well be a barrier where improvement of the model stops and completely new strategies must be devised.
Sure, you would need significant investments to start such a venture and success is not guaranteed. It is nothing a garage shop could do.

smoldesu · Answer

It's not hopeless, but it requires so much money and time that you might as well surrender all hope. Competing with OpenAI is tough, finetuning llama to a point that it competed with GPT3.5 required several finetuning attempts and many months of research.There's still room for more AI models, but competitive ones are remarkably difficult to build. Competing with OpenAI's thousands of GPUs is not an easy feat.

nutanc · Answer

https://twitter.com/RajanAnandan/status/1666641010284449792?...