On the open-source side, we had some low quality models and there was so much experimentation to see what works and what doesn't. Soon we got llama and llama 2 and a plethora of models to play with. So many projects started around them, many got abandoned shortly.
Now that I reflect on what happened in 2023, despite all the progress that was made, I feel like the pace of growth has decreased. In early 2023 I legit thought "this is how singularity feels like; every 2-3 days we'll have something new and exciting".
But now things are more settled. We still get new models every day and Mixtral is still amazing. But something seems off. No company (not even Google) was able to make something better than GPT-4. So many Chat UI and wrapper projects are abandoned (Github can be a scary place...), and we're not much further in our way to understand what the heck happens in these models than we were a year ago. In addition, it's become clear that GPT-4-level intelligence might be the best we can extract from the current LLM technology, and no one takes AGI seriously anymore.
Edit: I should add that Langchain, LlamaIndex and so many other "frameworks" built around LLMs that used to be all the rage now are evidently useless in production. RAG is still RAG, and no matter the tricks you play to make it "smarter", it's still just RAG. Langchain and similar frameworks cause more problems than they solve, and the tech debt is horrible. Vector databases are the same. Most are fighting for that sweet VC money, and their features are essentially the same. So many startups suddenly went out of business after OpenAI's first DevDay; so many more will perish after the second DevDay. It's unclear how $$$ VC will be allocated in 2024 given that the safest bet to make money off of AI was and still is OpenAI, not Google, not these third-party frameworks and libraries, not some wrapper around OpenAI's API with a nice UI and shiny website.
What do you think about all this?
“AGI” was just another obvious narrative to part VCs from their money to build more highly unprofitable startups once again. OpenAI being the ring-leader in this and executing this hype with an overvaluation of $100B for investors to purchase shares at that valuation. This is even before much larger risks such as the competition catching up quickly (Yes they are), mounting lawsuits and the costs / difficulty of training new models.
Reminds me of the Stripe hype with its valuation crashing more than 50% from the peak of $95B. Same with Klarna from the top of $45B with an 80% correction and I bet that Clubhouse is due for a >80% devaulation from $4B.
At least Midjourney was bootstrapped by the founders and making $200M+ without involving VCs with a small team. So I guess that justifies a much higher valuation >$10B and I assume it is also highly profitable.
GPT-4 Turbo, which was only released in November, is better than GPT-4 while having a much larger context. Don't be fooled by the fact that it's still called the same in marketing terms. It's a significant step forward, while coming only six months after GPT-4.
And "not even Google" is a strange phrase. Google hasn't been at the forefront of innovation anywhere in a long, long time.
> it's become clear that GPT-4-level intelligence might be the best we can extract from the current LLM technology
That's not clear at all. Where do you get this idea?
> and no one takes AGI seriously anymore.
Bollocks. LLMs are beating lawyers, doctors, and software engineers at post-university exams. We are galloping towards AGI.
What also happened in 2023 is that more than half a million H100s were installed globally, which are currently training the next-gen models you are guaranteed to hear about in the coming months. Compute capability is still growing at a stupendous pace, and there is no reason to expect that this won't lead to much more capable models.
I'm quite convinced that the only thing holding GPT4 back is the inference cost and context size. If we ever hit the point where inference is the same cost as loading a website, that's where I think LLMs will touch every aspect of daily life.
I have a lot of ideas using LLMs that are being held back by the above constraints.