Why can't Nvidia compete in the inference market?

Question

Hey HN! Lately I have been thinking about how the inference market will involve. We saw that llama 3 with groq was showing great results. Once question I had was why can't nvidia compete in the inference market? In the video below chamath was saying how nvida won't be able to compete in inference. https://youtu.be/1ZQ33OnGFWE?t=1254 is here just chilling his bag or are there some truth to this? What are fundamental reason that this would be the case?

talldayo · Accepted Answer

Likely because they don't make inference accelerators, they make GPUs. It's entirely plausible that someone could optimize the hardware pipeline better by incorporating less complex logic; I wouldn't expect Nvidia to try that though, not with their main products.

st380752143 · Answer

This reminds me of a sentence Jensen said internally: "We do the thing that only we can do"