In the latest Lex Friedman interview [1], Sam Altman said he expects (hopes for?) a similar level of improvement in GPT5 as the improvement of GPT4 over GPT3. He said GPT5 "just feels smarter" overall.
What do you expect, or hope for, to see in the next big model from OpenAI? What do you think its impact will be on the world?
[1] https://www.youtube.com/watch?v=jvqFAi7vkBc
I keep hearing this, "if you showed GPT4 to people 10 years ago they would think it was AGI!". What complete rubbish. What exactly do you think has happened over 10 years so that you don't have that response? Nothing really. LLM's were released and you gradually understood how they worked and (hopefully) quickly came to the conclusion that they aren't AGI.
Some people thought GPT4 was AGI when it was released, so of course some people would have had the same initial reaction 10 years ago. But just like today, reasonably intelligent people would quite quickly conclude that there was no AGI... Why on earth would you think it would be any different?
The goal posts for what constitutes AGI have not changed but many, many decades.
Unfortunately GPT-4 and even other coding co-pilots like Phind kinda suck at anything beyond "noob tier" / beginner level questions. In fact it has gotten so bad that I ended up Open AI subscription.
It falls apart at anything with complexity, and I feel that just taking out a sheet of printer paper or going to the whiteboard is always far more effective than trying to get GPT to come up with something that at the bare minimum has no errors.
...not really. GPT4 is just MoE so its good at more things, but its not really a huge upgrade over 3.5, the quality of which is directly a result of much larger parameter scaling.
I expect GPT 5 to be just more multi modal, perhaps with some added capabilities through prompt engineering internally. For example, when asked a question about code, it would execute the code, and ask itself if the answer is what it would expect or are there any issues. Or do things like automated web searching and using the html in the context, and provide answers based on that. And for any of those modes, there is countless optimizations you can do.
The thing about LLMs is that they are basically just the next Google, i.e efficient information lookup. There are 2 main things missing from them being able to reason.
1. Self guided recurrent loops. Rnns are difficult to train correctly, which is why transformer architecture took over, but for reasoning there needs to be some sort of recurrent loop that is self determined by the model to iterate on the correct answer.
2. Information ingestion without need for training loops, which looks like some sort of short term memory. Context windows aren't it. There needs to be some way for an LLM to look at a piece of text and with minimal passes auto configure the parameters to "remember" that piece of text. First is
Imagine you could load up that context window with information and it could use every bit of it with the same intelligence as if you only passed it a paragraph.
For example, function calling is good to maybe ten different functions before it starts forgetting them or mixing then up today. Maybe in GPT-5 we can pass in 1,000s of functions and it would be able to find the relevant one every time.
Or maybe we can pass in a huge codebase with line numbers, and it can identify exactly which lines need to be changed or added to implement a feature.
I hope it will be better in programming too. It does beginner mistakes, that can be fixed with one snarky comment, but that would be better to not do them.