How can I survive being targeted like this? Cycling IP addresses and updating DNS ad infinitum does not seem like viable solution... because in the end, it's the network attacks that get you, not the application level attacks.
What is your business? How much traffic "normal" do you get? What is the size of the attack? What is the bandwidth of your upstream connection? Who are your customers? Where is it hosted? What are your acceptable thresholds for false negatives and positives? Do you know who is attacking you and why?
Most every hosting provider will have some sort of DDoS monitoring and mitigation on their networks already. Their response to sustained or repeated attacks might range from scrubbing the bad traffic before it gets to you and not notifying you at all, to reaching out to you to work with you on both ends of the issue, to cancelling your account.
If you just have a fairly simple website that you host yourself, Cloudflare likely a fine option. If you have more advanced needs, you should talk to a more comprehensive DDoS solution vendor.
Application servers that aren't listed on DNS and their IPs kept secret. A _lot_ of proxy servers that take requests, check the auth, and forward the valid requests to the application servers. For invalid requests, just let them time out.
If checking the auth is expensive enough to cause problems, require some proof-of-work in the request too, and only check the auth if the PoW is there.
In other words, keep all the DDOS traffic off the application servers. Have sacrificial proxies. Make it cheap to handle the DDOS traffic and expensive to generate the DDOS traffic. And then just process the traffic. Spread the proxies across multiple cloud providers if you need to - but ideally ones with free ingress.
https://www.cloudflare.com/ddos/
Don't forget to have a fresh IP before doing the move so that your origin is unknown.
That probably depends on who your customers are. I can chime in one one scenario but maybe this does not help your case. If you are B2B then you could just shut off all public IP's and set up some Anycast VPN WAN accelerated meshes with your customers and use a wildcard sub-domain for each customer to access. They won't like having to set that up on their end but then they will have more resilience, privacy and security than they had before in that they can route around internet problems that would have otherwise caused outages or degraded services. Your company would have to provide professional services to help them with the setup and the customers would need discounts to offset the costs of the hardware assuming they do not already have it. The customers end-user devices would join their network over their VPN, not yours. For some B2B arrangements this isn't even a rounding error and removes scrubbing center costs and removes some SLA breach credits. Give this text to your principal network engineers, approve the spend and they will know what to do.
For the customers that do not accept this option, update the contracts and SLA's, then give them a public end-point that has a "best effort" SLA or use a cheap CDN if your contracts permit sending your sensitive data through them. Incentivize them to move to your Anycast VPN WAN accelerated mesh with tighter SLA's and improved security posture by keeping sensitive data off third party data processors like CDN's.
[Edit] This would also be a good time to use one of your companies sub-divisions or set up a new legal entity and get a new AS number with some dedicated IPv6 space and let your attackers just keep attacking the old address space. Announce the old IPv4 addresses on a throw away circuit to gather stats for law enforcement and network mailing lists. Try to do as much mesh peering with that new IPv6 space as you can. This is not security through obscurity so to speak, but rather a small piece of The Art of War.
The marketing team will not be happy but they can distribute their sites to a myriad of CDN's on marketing specific domains for your company. Gotta keep 'em separated, set some smart cache control headers and hope for the best.
An alternative to this semi-complex setup is to pay DDoS scrubbing centers to handle the routing of your datacenter traffic but that can hurt latency sensitive applications. This may be useful for the occasional extortion attack but would be very expensive and sub-optimial(c) in a never-ending indefinite DDoS attack.
You've mentioned that your hosting provider has DDOS protection (which is usually Voxility[3]/Path[4]) will usually send traffic to your VPS through their scrubbing centers, taking care of the L4 attacks. Ratelimiting and aggressive caching can take care of the rest.
If you're completely into this on your own, consider setting up a number of PoPs which relay traffic to your backend servers, and these PoPs should drop UDP traffic dropped the prerouting level to drop packets without high CPU consumption. There's also eBPF/XDP, but those require much more engineering work and a compatible network card that can run a eBPF program, otherwise you're still using the CPU to drop packets.
[4] https://path.net
There was HN thread asking the same question [2]
Having said this, none of my services were targetted by ddos, it made me thinking that maybe maintaining a list of 'healthy' clients IP addresses during good time and then banning anything from outside of that list in bad times? Sounds lame I know, I wait for ideas emerging in this thread :)
DDoS protection for web applications is a different beast than raw transport.
If you're a vicim of an amplified DDoS attack, your filtering may need to occur off the server.
It's possible that this doesn't exist for good reasons though, I am not sure how the numbers work out in terms of relative capacity.
A colleague was DDoS attacked and was able to move the service behind Cloud Armor in a few hours, effectively ending a ridiculous barrage of network requests. I think CA charges a minuscule amount per request, and can be capped with a fix monthly cost which limits the DDoS liability.
Consider migrating your website to web hosting providers renowned for their DDoS protection, such as OVH or Path. By doing so, you can ensure that your server performance remains unaffected even during ongoing attacks.
You can enable bot-mitigation and reCaptchas which will help a lot