So.. what's next after LLMs?

First we had deep learning - i.e fully connected layers with some data conditioning in between. This kicked off the current ML ecosystem, when people figured out stuff like auto encoders.

Then we had RNN, which have the output fed back into the input. This gave the networks some form of memory.

Then we had Transformers, which are basically parallel processors. I.e generate 3 outputs in parallel, multiply them together. This was basically just a better form of compression applicable to everything.

The general trend here is that someone discovers some architecture that works out nicely and then everyone builds something around it. This is probably going to be the future. Google has some neat things with automated robotics, OpenAi has their A* stuff thats supposed to be "accurate" instead of probabilistic.

Then there is the hardware piece, which I know much less about, but hoping companies like Tinycorp or Tenstorrent give us a way to reliably run something like GPT3 full parameter model at home.

My guess is the next big AI product will be local/edge and always-running (can initiate conversation passively). I also think deeply integrated AI into non-conversational software (again, local AI) capable of understanding and working within the embedded domain will become more widespread.

My wishful thinking is that instead of the corporate lead "ai evolution" some breakthrough and leak turns it into an "ai revolution" where a free linux OS comes with absolute privacy enforcement, ad busting in all forms of media and sites, automatic dark pattern removing web browser, untracable piracy, mass scale detection and highlighting of country trends in censorship / fake news / authoritarianism. And that it puts a ton of new pressure on companies having to have some sort of publicly proven morals

I really hope something more transparent, understandable and debuggable.

Also preferably something that wouldn't trigger the third contest about who can waste the most energy (after first crypto's proof-of-work insanity and then trillion-parameter models that needed a datacenter full of H100 GPUs just for training) while the climate change trajectory is already becoming more and more dire and we all really ought to reduce energy usage as much as possible.

(Edit: I forgot a third relevant point)

I am not sure I get exactly what you are pointing to, but synthetic logical answers are:

-- the lucidity after the fever, and

-- striving to implement what was missed. Plus,

-- understanding why what somehow works does work, and build knowledge on that.

For the first: LLMs will be placed in dangerous places as flexible but improper frameworks as a practice that will be deprecated by the community, and will in parallel be implemented in the places where they actually fit (we discussed a few possibility yesterday in these pages, for example).

For the second: the implementation of artificial morons (some handfuls of months ago) makes the "real thing" - something that reasons, that thinks organically, that criticizes its own contents, that is reliable - more missed, so more investment will be done towards that progress.

For the third: research will continue exploring why LLMs can produce surprising results (that some have linked to "scale"); knowledge built in this effort will eventually lead to progress.

Meanwhile, real needs will be tackled by businesses.

(If you can clarify what I missed in the question, please do.)

I agree with your assessment that a lot of this stuff is BPO. There is only so much customer service that can be automated. That being said, AI is a big tent and has been around for decades, despite the media claiming that OpenAI has been the first company to take it into the mainstream. AI flies fighter jets without any human pilots. I personally think there is already a lot of impressive stuff happening with reinforcement learning and am excited to see where large action models go, despite rabbit's failures.

It's possible that our next generation advances will be made inside, for example, bioengineering. I'm not super well versed on the topic so this is very speculative, but I've read a lot of cool stuff about DNA-based data storage solutions, and the thought now comes to me that may be that maybe lab-grown brains could achieve some interesting tasks.

Nobody can predict the future, but what is currently researched: - coupling deep learning with some kind of memory

- non generative hierarchical architecture (see LeCun JEPA)

- mixing deep learning with symbolic methods (usually search over token or latent space, it can also be a coupling with symbolic engines. Coupling LLMs with wolfram was a very early example)

- ability to perform unbounded computation per token inside a deep network. Something like Universal Transformers

What's next is real apps using LLMs efficiently. It's weird but besides chatGPT I see no game changing apps.

Probably LMMs (Large Multimodal Models) maybe?

I am not entirely sure if this'll be the future, but having one model that has relatively good performance over a huge range of types of data feels pretty good tbh

Marshal Brain wrote this about 20 years ago…

https://marshallbrain.com/manna

Humanoid robotics and "smart" glasses are the most intriguing to me to own on a personal level

Trust/truth apocalypse is the most concerning near-term

Hopefully not my long term unemployment, hopefully.

I’m looking forward to services that provide good value using the essentials - as an alternative of the deeply enshittificated by LLMs versions. I’m already looking at Kagi for example.

Color me skeptical but the probabilistic nature of LLMs is at their core and is a hard limit to how useful they can be in wide applications. Currently their input and influence are limited to bulshitting - useful for mass cheap propaganda, seo spam and writing soulless student essays or slightly wrong boilerplate code.

In an informational landscape that is the equivalent of an infinite garbage dump, purity becomes priceless — the new unobtanium.

I think we’ll be on this train for a while and it’ll just scale insanely far ( think photonics etc)

After LLMs? I prefer a better, more accurate LLM.