Why was IPv6 a significant redesign? (Rather than being a “longer IPv4”)
... while BGP done a much more seamless transition on AS numbers?
Partially, the tail of things dependending on IP is a bunch larger (i.e. AS numbers are a concern to BGP only, IPs get handled by lots of other protocols). But also a massive dose of trying to make a big new standard with everyone involved, which lead to all kinds of grand ideas to make things more elegant and enable great new ideas - and a lot of them turned out to be ideas based on assumptions that just don't hold anymore. (E.g. a bunch of the extensions and routing stuff assumes that routing is done by general-purpose CPUs that have time to look at and fiddle with every packet. But of course that quickly stopped to be a thing)
It's very rare to make a breaking change in a protocol as ubiquitous as IP. One thing about making a breaking change is that once you've committed to one such change it becomes much cheaper to bundle in other changes at the same time. Extending the routable address space required a breaking change and everyone recognized that it was likely to be the last such change for a very long time. This meant there was immense pressure to resolve any shortcomings in IPv4 and introduce new refinements to take advantage of this one, possible last, opportunity to make breaking changes.