Objective Function for AGI

Question

The objective function of LLMs are to predict the next few tokens after in the best possible way.But for humans, the objective function is survival. Evolution has created us human beings, and evolution works by survival of the fittest. So the objective is survival.How do we digitize this objective of survival, so that it can be used to train models leading to AGI?

JoshTriplett · Accepted Answer

1) "survival" is a massively oversimplified view of human value functions. Evolutionary biology leads to "successfully replicate genes", but humans are a lot more interesting than that precisely because we can choose to have much more interesting value functions. Anyone and anything that's attempting to actually understand humans needs a much more nuanced view.2) Don't. It's incredibly hard to build a correct value function for an AGI that won't unintentionally kill everyone. "Successfully replicate" is precisely the kind of thing to avoid. That's directly the "gray goo" scenario: https://en.wikipedia.org/wiki/Gray_goo