Normal tools are focussed on one speaker at time.
Zoom supports breakout rooms, but they are a bit clunky.
I'd imagine something where you can "move closer" to certain speakers which would increase their volume, but still listen in to others.
Your "position" would be visible to everyone, so you could move around in the room and leave/join dicussions more natural.
This would be a replacement for we are all having lunch in the same room / company event.
Reminds me of second life ...
right now on Show HN: https://news.ycombinator.com/item?id=22772800
Mozilla has something like this as a "VR" demo: https://hubs.mozilla.com/
Mumble in combination with a game like Minecraft can do it.
Not aware of anything more finished.
without: Jitsi at least makes creating new rooms frictionless, but would need an extension to show an overview of who is where.
EDIT: since you mentioned it, I guess Second Life still exists