Is this a solved problem?
If you need to scale more (a lot more) or if connectivity is a challenge (kind of inexcusable given what kit is available), then you need to stop trying to unicast your stream wide & embrace multicast. If it's a single audio stream you dont need to fight the base rate war, per se, but if the base rate is 1mbps you are asking for a lot of airtime to be uninterrupted for really shitty data rates. I dont know what kind of smarts modern APs have here, but generally I see little value trying to low base rate punks- maybe your mercy grants them the ability to connect but if the connectiom is that bad they probably shouldnt be allowed to connect. Telling the AP to use 6mbps, 12mbps, or 24mbps rates for base rate will vastly shorten your transmission time, & make reliability higherr, at the cost of kicking out some fringe devices.
Advice to configurr base rate (which definitely affects multicast in wifi) has been around for multiple decades & is unremarkable. Players that can compensate their playing to a "global" clock are unremarkable.
Make one device as teacher and all other devices as students
It's a bit janky, but VLC Player can act as both the client and server, here. Shoutcast did multicast, too, the last I checked.