What is meant by “AI experts don't understand AI”

They don't understand as in OpenAI learned that ChatGPT is capable of doing things it was never intentionally trained to do after it was released. I've heard anecdotally that self-driving cars can pick up the unwanted behavior of inching up to a red light in anticipation of it turning green and they were never trained to do that specifically.

> Neural networks find features via gradient descent so humans don't have to encode all of the features.

Yes, but what are those features? What did it decide was important? Often, we aren't really sure. We often can't ask the model to justify its decision.

It means that these AI systems are essentially unexplainable black-box systems that not even AI experts can figure out why it made those decisions. Example: (Self-driving cars confusing the moon with a yellow light, ChatGPT making up broken code or unsafe legal advice.)

This is why it is very difficult for anyone to completely trust these AI systems in very high risk situations. You would not board an Airbus jumbo jet without any human pilots and only controlled by an AI to fly from A-to-B end-to-end. (Not the same thing as autopilot)

Just ask any of these Turing award winning AI experts (Hinton, LeCun and Bengio) and not even they know why these systems hallucinate given the opaqueness of these AI models.

> isn't that the whole point of AI

No. A feature of Artificial Neural Networks is automating function approximation. To not understand the output, the virtual function and its approximation, is not a feature but an issue.

It has been called "the problem of transparency" by some authors.

Artificial Intelligence, in general, is the construction of automated problem solvers and can be (historically has been largely) deterministic.