Criteria isn't strict, but no devices that are just an Android with the cellular modem taken out, which I see a lot of listings for. Rockbox support is preferred but I've had no luck finding anything in their supported devices list that's not an iPod, used or new. I'll probably get a used iPod if nobody has an alternative recommendation.
iOS has KMPlayer which was good the last time I used it: https://apps.apple.com/us/app/kmplayer/id835776444
Also Plexamp if you want to use Plex.
Walmart has a big selection of mp3 players: https://www.walmart.com/search?q=mp3%20player&typeahead=mp3%... However, i was not able to find good enough one that I would recommend. You can try searching yourself.
I personally use Plexamp and Musi.
BS19C Bluetooth-Compatible Headset Sports Music Headphone TF Slot FM Radio
iPods are still a really awesome device for pure music listening in my opinion. There are a lot of mod options to bring high-capacity modern storage to them, even Bluetooth and haptics. Lots of good info in the sub:
What an absolutely abominable state of personal computing.
Don't they have itunes? Or is it not good?
You need to run a Plex server I believe.
But is great and works like you want an MP3 player to work!