Is Apple's R1 a Discrete GPU?

The R1 is an SPU (Sensor Processing Unit) which works in concert with the M2's integrated GPU (the R1 being the innie, and the M2 being the outie). It is doing an enormous amount of image signal processing, which seems to explain the 256GB/s memory bandwidth.

https://techcrunch.com/2023/06/05/apple-r1-chip-apple-vision...: "The specialized chip was designed specifically for the challenging task of real-time sensor processing, taking the input from 12 cameras, five sensors (including a lidar sensor!) and six microphones. The company claims it can process the sensor data within 12 milliseconds — eight times faster than the blink of an eye — and says this will dramatically reduce the motion sickness plaguing many other AR/VR systems."

Maybe instead of speculating based on Apple's marketing and based on articles from other people speculating, we can make some guesses from actual data.

Here's an obviously-incomplete list of source files part of the R1 firmware, probably referenced from asserts or other logging messages, thus present as strings in the firmware binary:

https://transfer.archivete.am/inline/Ydfxb/bora.txt

It seems it's handling data from cameras (CImageSensor*), LIDAR (SensorMgr/Tof = time of flight?), and display (DCP). I also see mentions of accel, gyro, bmi284 (IMU from Bosch?).

I'm very interested in it as well, I think we can mostly just speculate. I don't think it's a GPU, I think it's an independent SOC running some sort of real-time OS (https://en.wikipedia.org/wiki/Real-time_operating_system) I suppose it would necessarily have GPU cores in it to do image processing, but I don't think it itself is just a GPU, or gives any indication about discrete GPU's coming to the Mac. I think this is a very Vision-specific piece of silicon.

Edit: many are saying that is just an Image Signal Processor (https://en.wikipedia.org/wiki/Image_processor). I don't think that's quite the case because 1) The M-series chips are already known to have ISP's packaged into them. and 2) My understanding is that the R1's job is to provide continuity of passthrough even in the event of a kernel panic by the M-series chip. To my thinking, this means the R1 chip must have a level of independence beyond that of a traditional coprocessor. I think it is an entire SOC.

I don't think it is a GPU, rather it is probably very similar to a DSP/Neural Network chip. Lots of matrix multiplication capabilities and streaming data.

https://en.wikipedia.org/wiki/Digital_signal_processor

(Of which https://en.wikipedia.org/wiki/Image_processor are a subset of, which someone else in this forum mentions.)

DSPs are similar to GPUs in many respects but they are also similar to CPUs in many respects, but it doesn't make them the same.

I thought it was fairly clear from Apple’s comments that it is essentially single function hardware that does all of the processing for AR/VR. I mean, they say that it is all done inside 12ms. There’s no way that it is just a GPGPU. Modern GPGPUs, particularly the NVIDIA stack, have a bunch of single function units internally which do things like optical flow on the image buffers etc. but similarity sort of stops there

You can do a lot with not much if it is all specialised hardware. Some of the wider features of the chip are due to the huge data bandwidth. But general purpose processing is a little slow for this task, certainly at this power envelope

“ Apple shows it as being roughly the same size as the M2 processor in the Vision Pro marketing[3], indicating it's a peer to the M2”

Errr, that’s not how this works. The size of them being similar means absolutely nothing.

The R1 is a coprocessor, I mean it’s basically a big signal processor with a decent amount of RAM to handle all the camera inputs, and does some GPU like stuff, but isn’t a GPU

I doubt it's a discrete GPU. It doesn't make sense to split a GPU and CPU in such a tiny device. Especially with the gains you can get with completely shared memory. I think it's much more likely to be a SoC with dsp tiles for the sensors and a huge GPU block and standard apple p and e cores.

A 1 Gigabit DRAM chip sounds far smaller than anything that's been used with GPUs in a very long time. That's a capacity from 15 years ago.

There is no way Apple designed the R1 from scratch, it will contain a bunch of components from their other SoC designs. Obviously it's going to have quite a few more IPUs than a standard SoC, which only have to handle ~4 cameras. It also clearly has the display scan-out components.

And it's entirely possible that one of the components Apple took from their grap-bag of SoC component is the GPU compute cores. Properly not the rasterizer or texture samplers, but I could see the compute cores being useful for running tracking algorithms. The massive amount of memory bandwidth does kind of suggest GPU compute cores.

But even if it does, that doesn't make it a discrete GPU. Just a dedicated SoC with some GPU components, and a bunch of other things.

But it simply can't be used as in indicator of discrete GPUs in Macs, because all those other SoC components would be a waste of silicon and it's unlikely Apple would reuse the R1 die as a dedicated GPU.

it’s a discrete Image Processing Unit

https://en.wikipedia.org/wiki/Image_processor

No, likely a dedicated DSP chip to handle the aforementioned 12 cameras + 5 sensors + 6 mics.

It's basically Apple's version of Microsoft's Holographic Processing Unit

It's proprietary magical undocumented locked-down tech that I don't want anything to do with, regardless of how well it performs.