How do you correct spatial reasoning of LLMs?

Question

I'm working on a complex multi-part 3D-printed product, and I needed some engineering input from LLMs. I use highest Gemini Pro reasoning levels, Fable, and K3 Max with everything tuned to highest.The problem I'm facing is mounting a sensor on a barbell sleeve. The sensor is a semi-hollow 100mm cylinder, 60mm diameter, with a 50mm diameter 50mm deep tube cutout for the sleeve. (It's a bit more complex than that.)I'm trying to brainstorm different clamping mechanisms, and evaluate durability and manufacturability of several designs.No matter what I try, the output of all LLMs is absolutely wild, especially if you ask to make a technical drawing. Anthropic reliably been the worst, and Gemini, surprisingly, the best.But none understand what goes where in any detail, besides spewing paragraph and paragraph of wild ideas including rotational cam clamps over a recessed ring TPU collet (apparently, a "brilliant pivot") and stuff like this.Is this the "unconquered frontier"? Do I need to blow dust off my AutoCAD? :)

spottedmarley · Accepted Answer

Everybody is working on (or waiting for) world models. Language models are not sufficient for doing work in the meatspace.

xyzzy123 · Answer

I would consider asking it to model its proposed designs in openscad or build123d (ideally something query-able). Then have it render and examine plausibility / suitability from different angles. Get it to render the part in use also and give instructions to think about forces and motion.
Recommend doing this in a coding harness not a chat box.
The reason I think you might have more success with this is that the model is mostly thinking about the part in words, which it can convert to a part design in CAD in code and LLMs are really good at coding. Also means it can use relative positioning and relationships.
You will be able to iterate more easily, compare things, compute properties, commit to git etc and the process is more steerable than generative production of images.
Once you can get renders most current models should be good enough to discriminate when they are producing nonsense, misaligned parts, things that don't fit etc. It's not going to be perfect, but I suspect it'll be much better. The whole process of code -> render -> inspect forces the model to put up or shut up and provides grounding.
As far as I know today's LLMs don't have a "visual imagination" but a process like this could be a slow approximation of one. They clearly do have SOME spatial understanding (pelican tests show us that!) but it feels really non-human.

krapp · Answer

It's wild to me that this is even a question, because of how AI brained the entire world has become. I mean, you have a tool that you know works, and a tool that you know doesn't, but you still feel the need to find a way to make the square peg fit into the round hole.
I'm sure there are ways to "correct the spatial reasoning of LLMs" (whatever that even means) but I'd question whether you even need an LLM to begin with. I know that isn't the answer you're looking for, I'm just a slack-jawed knuckle-dragging Luddite. But this seems like the kind of simple, common problem for which established solutions already existed, like "talk to an actual engineer and not a chatbot that can't even do basic math."
And if it's a matter of not wanting to pay someone, consider that if you're already paying for an inadequate simulation of knowledge, you should be willing to pay for actual knowledge.