I have also read instances where LLMs are used as eyes and render the environment on a virtual canvas as they "see it", they are simulating the operations of a complex biological organ and sense.
I'm somewhat reminded of the paper "Could a Neuroscientist Understand a Microprocessor?" which calls out the weakness of black-box analysis and simulation techniques. https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
https://xuwd11.github.io/am207/wiki/functorch.html#:~:text=A....