Why is there not more concern about the physical security of Cloudflare?

It's interesting to explore https://where.durableobjects.live/ - a tool that maps where Cloudflare's worker scripts actually run.

Notably, while Cloudflare has CDN edge locations in countries like China and Russia they don't appear to run workers there.

EDIT: I was wrong - I misinterpreted the map. A solid border circle around a location indicates "Worker-only Datacenter" (see the map legend) and there are indeed locations with those solid borders in Russia (including Moscow and Yekaterinburg) and China (Haidong, Lanzhou and more).

I doubt we could get them on the record for this, but I suspect this may be very deliberate. Maybe CDN edge locations can be run completely securely with forwarded encrypted traffic, while workers are at a higher risk of physical attack.

Cloudflare uses AMD TSME to encrypt the RAM:

https://blog.cloudflare.com/securing-memory-at-epyc-scale/

What's the threat model that ram interception is an issue? I think the upsell is entirely reasonable, you get charged more for weird compliance demands.

On Azure -- they've been playing catch up this year after repeated congressional inquiries from breaches. It's only in 2024 that Azure has started to build a better device inventory on their infrastructure networks and started doing appropriate employee access control mechanisms.

Here is Satya's May Post, https://blogs.microsoft.com/blog/2024/05/03/prioritizing-sec...

Isn't this basically why modern server CPUs support Secure Boot and memory encryption? If I understand it correctly, it should be possible to set up your machines in such a way that 1) the server can only boot genuine firmware images, and 2) unencrypted data is only available inside the CPU itself. That is going to rule out most attacks which don't involve dragging the server into a high-end laboratory, is it not?

They have some docs on the security of their server hardware, which is one of the ways they can assure that their systems haven't been tampered with by local staff.

https://blog.cloudflare.com/anchoring-trust-a-hardware-secur...

Why not use your own datacenter if it’s that important to ensure physical security? If the business can’t justify it, then maybe your use case isn’t that sensitive or important. Or maybe the business isn’t viable.

Curious how they protect private keys for transit encryption. I'd imagine all their edge locations need to be able to decrypt traffic and therefor need a way to fetch corresponding private keys for everything they're proxying.

If physical theft is a concern, how do they prevent someone from hijacking the key distribution process?

I don't use Cloudflare and I don't have strong opinions for or against them. These questions are the same I'd ask if you were discussing any other company:

> which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.

Why do you say that? Do you have signals that their colo facilities are less secure than they should be, and/or that Cloudflare hasn't gotten those facilities to beef up their security as part of their contract? Again, not saying this to defend Cloudflare. I just hadn't heard this before.

> Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings.

That's going to be the case with almost all providers in the space. If you're asking for special treatment, you're going to have to pay for it. I don't mean that to insult you. At a past job, for various reasons we had strict compliance obligations that our data could not be accessed outside of the US. Some of our vendors used offshore tech support who'd have access to our data, and a couple times we faced a decision: pay that vendor $$$ to special-case our support setup to meet our requirements, or choose another vendor.

> Cloudflare has also deliberately restricted our ability to block non-Enterprise Workers, KV, and R2 from specific regions, leaving us with limited control over where our data is processed.

Same. Those fine-grained controls are often going to be an enterprise feature.

Again, I'm not saying this to defend Cloudflare in particular. They have their own paid spokespeople. I'm not one. Nothing you've said sounds particularly egregious though. If your data is sensitive enough that you're legitimately worried about someone sneaking in undetected and intercepting RAM, prepare to pay the enterprise tax with all your cloud vendors.

ultimately you trust the company and the jurisdiction in which they operate...if they give you the wrong answers then you should adjust your level of trust accordingly...thankfully there are other platforms and this is one concern that can certainly be better marketed

There are, roughly, four kinds of physical security one can have in a datacenter:

1) Nothing

2) Visitor logs

3) Locks and alarms on your racks, and/or (if you have enough) the rooms they are in. Remote monitoring is pretty common.

4) First party human security professionals

I don't have any special knowledge of Cloudflare's set-up, but 2 and 3 are by far the most common. Lacking #1 means your hardware just gets stolen. #4 is too expensive. #2 and #3 are where most people are, so probably something around there?

Sounds like you want an elevated level of security that cloudflare provides but you don’t want to purchase?

Cloudflare have some really interesting tech, like https://blog.cloudflare.com/announcing-keyless-ssl-all-the-b..., that I really wish we had an open implementation of so I could do similar myself.

> which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.

what does this mean?

What about other providers like Fastly and Akamai? I've not looked into this but maybe they do a better job?

> Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings

so you want the cheap plan with the features from the premium plan?

never heard of a story where physical security at any cloud provider has been a problem. are you worried about governments, or employees, or someone breaking in?

Some of my own thoughts about this. I don't know, I don't work at CloudFlare. Anyway:

> infrastructure, which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards

What are these questionable facilities? When CloudFlare has installed those servers and software, and they have also made software that manages those servers, what is the problem?

CloudFlare has written some articles about some of those very many security protections they have. But that is a very lot of technical detail to explain, so if that is so important, their Enterprise/FedRAMP offerings fund CloudFlare to make possible to explain that amount of detail. But question is, do you really need that amount of detail? How much you have expertise to build same amount of security protections? Isn't it better to use CloudFlare Workers security features to concentrate on building your app? Alternative is to get your own bare metal servers, and manage them yourself.

With CloudFlare Workers, they have security features to keep code and data of each customer separate from each other.