I don't think that's just a joke. There is a finite amount of space in your skull, so you can have a finite amount of neurons with a finite amount of interconnections and so on.
Some people fill up faster than other people do. You've done quite well compared to normal people.
The biology is even biased toward early learning. Humans have an unusually high ability to keep learning into adulthood, but it's still nothing compared to childhood ability. Learning tapers off in favor of stored knowledge. The elders hold knowledge and have less urge to shake things up, having already figured things out and obtained status.