Part of it seemed to be that I was querying an anycast server, the Google DNS server 8.8.8.8, which is probably many servers. So I wasn't hitting a cached entry because I was hitting a new server.
There are cache invalidation processes, the BIND name server has had one for ~20 years between authoritative servers. Google and Cloud Flare provide mechanisms for you to submit your own invalidation requests on a zone, and one could imagine a big provider like AWS doing that.
See the discussions in the above thread for more links to how to test and invalidate.
If you provide more information about the type of DNS server, who controls it, how it is managed and any default TTLs, perhaps people can help explain the shorter propagation period.