One of our Azure accounts was hacked – how to negotiate the bill?

Step 1: Read your cloud services contract with Microsoft very carefully. What does it say about your liability for fraud?

Step 2: Read your business insurance policy very carefully. What does it say about fraud coverage? What are the limits and exclusions?

Step 3: Unless 1 or 2 makes it real clear the business is not liable, get a lawyer.

If not done already, prepare off-cloud backup and consider migration plans.

There is some risk that they will terminate your account.

That really sucks and is the risk of using cloud solutions with no spending limit and a lack of monitoring.

You should still have someone to keep an eye on it when using cloud solutions. And when you already have someone to keep an eye on it there's a good chance you might be better off managing the infrastructure yourself.

>but I think it's more like a stolen credit card situation

How did the account get compromised? What was the nature of the attack (e.g. cryptocurrency mining, expensive egress traffic for file hosting, etc.)?

Every (consumer) credit card I've seen requires you to take reasonable steps to keep the cards secure to be eligible for fraud protection (e.g. changing the PIN if compromised, not lending it to people, alerting the issuer ASAP in case of suspected fraud, etc.). I do not use Azure but I would imagine that it works the same way - that is, if you fail to follow basic security precautions (enabling MFA, not using shared accounts or passwords that have been known to be compromised in a leak, etc.) you'll probably end up stuck with the bill. Hopefully you had things reasonably well secured.

This is a conundrum. On one hand, I understand how frustrating something like this can be. But on the other hand, your cloud provider did provide those services that you're being billed for. So they did incur costs, why would they just eat those costs?

Unless they're somehow at fault by exposing your credentials or making it easier for hackers to log in without 2FA or something of that nature.

If you're using a credit card to pay (though can't see a credit card having a 200k limit, even business) you might want to see if they can help (though it's not the credit card itself that was stolen, so it's unlikely they'd cover you). Otherwise, I'd imagine you're SOL unless you have some other insurance you can rely on.

I’m sorry to hear this, this is a tough situation.

Microsoft might, but are unlikely, to help you out.

Similar situation with your bank. Neither face a legal obligation to help you, just potential bad PR if they don’t.

Your best bet may be bankruptcy. It sounds terrible, but assuming you have an LLC/Ltd company, you can clear out your coffers, wind up, pay them pennies on the dollar, if anything, and start a new business. You may need to go through an lawyer or administrator depending on bankruptcy laws where you are.

I’ve taken a client through this, after a similar situation - they ended up with a vast bill to a supplier brought about by someone else using their credentials, and the supplier not being willing to budge. It cost about a week of time and about $2k in legal fees.

I’ve also been on the receiving end, where I presented a legitimate invoice and rather than pay the client reincorporated and kept the IP - which sucks, but Microsoft will be insured against insolvencies, so I wouldn’t feel bad about it. You’re just allowing their insurer to help everyone out.

Please setup billing alerts, know what your daily spend should be, add a little for if things grow a little unexpectedly. But you should absolutely be getting alerts if your spend is out of the ordinary for > 2 hours.

Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim.

Once you pay it, you lose all leverage. You're much less likely to ever get any money back.

Probably consult with a lawyer.

Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In my experience, companies are pretty willing to forgive fraudulent charges if you don't have an unusual history of them.

Try to contact Microsoft support immediately.

Don't rely on the distributor/vendor, they act very slowly.

You're a customer of Azure, you can contact them by any mean, the fact you pay through a distributor doesn't change that relationship.

So I would open a Azure support, and also will try to find Azure team on Twitter/Hacker News etc and contact them politely for help.

There is no way you would have to pay this bill. They will sort out something or even waived it if it's the first time.

That's unfortunate situation. It happened to me once before (though, we was using AWS that time. And, I believe the cost was smaller than the one you have right now).

What we did to recover the cost was to contact the account manager for our region at the time. So, maybe you could have better luck trying to find the particular person in linkedin. Or, have you tried opened a ticket from Azure console?

Nonetheless, I hope after everything has been settled down, you won't fire anyone (and treat it as learning opportunity)

I'm quite surprised that there isn't some kind of monthly budget control. For every new project I set the budget to be 4-5x my expected expense.

I asked microsoft support to vaive the past two months of billing because I left open a database cluster which I created for testing purposes. They promptly replied, took me through the steps and vaived the bills.

So maybe just file a support ticket, or have your distributor file a ticket for you?

When that happened to us, we found an article showing Tesla got hacked the same week as us (was aws) and they got the money back, so why not us?

We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone.

I'm not sure why you would be liable for fraud

I wonder if you can take out an insurance policy against this. Many of them have cyber-fraud coverage… perhaps this would qualify.

AWS would reimburse this if it was the first time. Maybe some hope for MS to do the same?

I can't help you with the legal side of things, but moving forward I advise hiring some security-aware infra guy. The root cause of most of these incidents is some human being incompetent (leading to things like poor security and relying on manual processes) or reckless.

You had no 2FA enabled?