What happened was that at some time around the end of 2019 the credit card expired and nobody noticed it. The API just kept on working for our services, so we assumed everything was fine and we were probably within the free tier. The card was added and account configured by someone not working on the project anymore.
Then recently the service just stopped working and after a bit of digging around, we found that the account had racked up a bill of $20.000 since the end of 2019. Slowly and steady each month. It was nice enough of Google to give us quite the credit period there, but as it turns out after inspecting the cost breakdown, someone else must have been using the key since a lot of the requests were to API endpoints totally unrelated to the services we provide.
We screwed up big time by having an unrestricted key configured. It was setup in 2017 way back when things were free, so I doubt anybody actually thought about abuse of the key at that time, but none the less, here we are having our key abused.
Do we even have a chance here? Did anyone get into a similar situation? I would be grateful for a little bit of advice about how to approach Google. I realize this is our fault, but at the same time, it seems we never got any payment reminders or notifications that we had bills overdue - at least we cannot find them anywhere.
A broken / blocked key should not result in this happening. It's a failure of the software. You should also be able to get the IPs that your account was connecting from, and disavow those that are not yours.
Divest yourself from Google and don't pay the bill if there's any way out of it.
I would negotiate for reduction based on the fact that you didn’t get any notifications.
I can help. How can i get in touch with you?
However, I'm quite shocked when you write "when things were free, so I doubt anybody actually thought about abuse of the key at that time" It look like - for you - having something free mean having something you can use without any care! From my point of view, if someone give me something for nothing, I will thank and try to use it wisely... But maybe I didn't understand what you meant