How physical server boots are done scale with disk encryption enabled?

Question

Just want to understand how physical server reboots are done at scale when the disks are encrypted and requires a boot password.When your data center has millions of devices, how this problem is solved at scale ? Any readily available tools ? Have a big operations team who will do only patching/rebooting ?

GauntletWizard · Accepted Answer

Very few servers at large-scale data warehouses have disk encryption of the form that requires a boot password. Most will have encryption on the data volumes, and even at the OS level will have some form of encryption where the key is stored next to data (So that the key can be wiped/changed to easily "wipe" the data)
To look at what companies like the FAANGs are doing, see how Hashicorp's Vault models threats[1]. When in High-Availability mode, physical presence of operators is only required for Sealing/Unsealing[2]. In essence - There is a cluster of servers that can lose members, and the other members retain access to the "Master Secret". Members can part and rejoin without the "Master Secret" being lost, and will automatically distribute the secret back to those that have rejoined, if they're still included in the cluster and have their own private key. These few servers do require manual rebooting to access their private key again, but there's enough redundancy in the cluster that they can simply be rebooted during normal operator hours.
The threat model is that you will not be able to break in to a datacenter and physically suborn a server before being caught, but you might be able to walk away with a server or plug in a new one with old leaked secrets.
[1] https://www.vaultproject.io/docs/internals/security/ [2] https://www.vaultproject.io/docs/concepts/seal/

devnullbyte · Answer

Network-Bound Disk Encryption (NBDE) using something like clevis / tang can be used or a sealing the luks passowd in a TPM.