There is some priority for example a device recently raised responds first - this is often your iPhone. The Mac is low on the priority - HomePods are usually highest. Amongst HomePods it's usually the one that hears the command first or at the loudest volume. You can remove your iPhone from the pool by placing it face down.
Devices can't be selective based on the audio because they're not analyzing the commands until they're actually activated. The always running Secure Enclave waits to match the voice pattern "Hey Siri" without the device knowing anything at that point. That's why Apple can say the devices are not listening or recording - they don't wake up and start listening until the Secure Enclave tells them to wake up. I believe however the devices keep a short amount of audio so that they can roll it back after they actually are activated.
I do agree with you in principle however that Apple should figure out some way for users to prioritize various devices. You can of course turn off "Hey Siri" responsiveness but there should be a bit more to it.
It doesn’t sound like it’s working well (or at least predictably) for you.
I keep it off on all but my watch (only on raise) and my HomePods. I trigger the other devices manually by holding the appropriate button.
If Siri worked better with between devices stuff, e.g. “start a 10 minute timer on my watch”, I wouldn’t have had to do that. Some things work like “start The Simpsons on Hulu on TV” but so many don’t.