Is not running a “dual stack” IPv4/IPv6 website costing me users? (yet)

I don't have a specific answer to your question but if you wanted to get some rough numbers based on your current visitors, create an A record and a different named AAAA record pointing to two images on your site, one being an IPv4 only image and one being an IPv6 only image. This assumes you still have IPv4 for now and are just contemplating dumping it.

    ipv4only  1h in a x.x.x.x
    ipv6only  1h in aaaa x:x:x:x:x:x:x:x

Then in your apex/root page lazy-load the two images ideally in a way that won't block rending the page if one of the images does not load. One image references the ipv4only domain name and the other image references the ip6only domain name. Both of those domains should be added as aliases to your main virtual host. People can also see visually which versions they can reach. Others here can probably suggest the best way to do this that will not affect the speed of your site. You can then parse your access logs to see who can load both, one or the other.

Your subject says "is it costing me users?", but your message says "how many people will be unable to access the site?".

These are two different things.

If your site is slow, it will cost you users. v6 has measurably better performance for websites, so not having it will cost you users. Those users will still be able to access your site though, they'll just choose not to because it's too slow for them.

The practical reality is that almost all client devices that are used for browsing the web will have some mechanism to reach legacy v4-only websites (but the mechanism might itself be a cause of poor performance...).

On the server side, v6-only servers (VPSs/containers/etc) are getting somewhat more common though, so if you're hosting something that's expected to be downloaded directly onto a server then please figure out v6 somehow.