How does your company manage its encryption keys?

Someone at my company generated the keys. They then put them on a network share without any security restrictions. They've been there for 5 years with no rotation. At least 2 are checked into source control.

https://www.vaultproject.io/

We use Hashicorp's Vault product to manage SSH credentials, TLS certificates, as well as application secrets across thousands of users, tens of thousands of virtual machines, and hundreds of applications.

We pay for the enterprise version, but the free version is more than capable for most needs. Avoid a password manager if you can, it leads to poor security practices and availability issues depending on how and where the data is stored (IMHO, YMMV). Use single sign on where ever possible.

Disclaimer: No affiliation other than a satisfied customer and frequent end user of the product.

Here's what works for small and medium organizations for data which needs to be encrypted at rest, but is not often accessed (so, backups):

1. Buy a bunch of Yubikeys, minimum of 2.

2. Create GPG keys and store them on YubiKeys. Follow this guide: https://github.com/drduh/YubiKey-Guide (if you want to, keep the secret keys, but in case of multiple YubiKeys I would not keep them anywhere). Remember to set the keys to always require touch.

3. Use GPG to encrypt your backups to multiple recipients (all of the YubiKeys).

4. Take care of the physical keys with proper storage and procedures. Do not store the keys together, have at least one in a really secure location, check if you have all the keys regularly, etc.

5. Test restores at least once per quarter, with a randomly selected key.

The advantages of this solution is that it is simple, works pretty well, and gets you a lot of mileage with relatively little inconvenience. You don't have the risk of keys being copied, and guarding physical keys is easier than digital ones.

You still have the problem of guarding the passphrases to the Yubikeys (if you use them), but that is much less of a problem than guarding the encryption keys. A passphrase without the physical key is useless.

This setup works for organization from size 1 up to fairly large ones.

Note that some recently fashionable security consultants crap on GPG from great height, but do not provide an alternative. It's a tool that while having multiple flaws, does many jobs better than anything else out there.

The more complex your system grows, the more often it will fail and shoot you in the foot. I'd advise against systems like Hashicorp Vault - they just increase the complexity and while they have their merits in complex setups, you seem to be too small to be able to operate such a system.

Have an offline backup printed along with the disaster recovery checklist and documentation and put them in a safe in your company - the checklist should be dumb enough that your drunk self can use it at four in the morning, because you were the nearest employee when everything went down.

Ensure that you have stupid manual processes in place on rotation of the safe's PIN and encryption keys in general, including a sanity check if the newly generated keys actually work (e.g. if they are used for your backup storage, actually back something up and restore it). Ensure that the safe's PIN is available to at least another person and used regularly (e.g. because you store your backup tapes there).

If you feel that you need to change from this very simple system to a more complex one, ask yourself why. What does your change actually add in terms of security and what risks does it add.

In the end, you want your system available to customers and the security you add is to not only secure the data, but actually to know who can access it (the auditing part).

Can't speak to my current employer as it's above my pay-grade to know, but at Job-1 we did the following:

- All "hot" keys were stored in an offline credential manager in specific vaults depending on who needed access to them. Only staff with actual clearance could request temporary access to a vault (fully background checked, 1 year employment, etc).

- Copies of each vaulth and our master CA cert were written to 4 encrypted USB sticks. Two stored on-site in the fire-safe and two off-site at our safety deposit box that only c-level staff could access. (We had the same process with our tokens and master logins for AWS).

- Any work using those keys was on a pair-up basis, so at least two people, one doing the work and the other observing.

- We had a detailed policy around this that covered each step in the process and who needs to approve them; everyone who could feasibly need to access the keys was briefed annually as part of our security awareness training.

We handled a LOT of sensitive financial data, so this was the most appropriate way that we could find that maintained both sensible availability and key control.

So in order to get to the keys you needed:

- Access to the fire safe (Senior Ops, Senior Security and C-Level only).

- The LUKS passphrase for the USB sticks (Senior Security and some C-Level only).

- The passphrase for the specific vault (Senior Security and some C-Level only).

I don't know how the passphrases were managed by our sec team, but I know that the C-Level staff had physical envelopes in their home safes.

YubiHSM 2 has worked fantastically well for us as a root of trust in a variety of applications, at a very reasonable price ($650) (I am unaffiliated with Yubico other than as a very satisfied customer).

Accessible over USB or HTTP, it supports every major crypto algorithm [1], and keys can be backed up onto another HSM via a wrap key (if they are marked as exportable -- you can also control what can and cannot be exported -- in fact, every operation may be allowed or disallowed per key).

Every operation is logged for audit, of course, and the device may be setup to require logs to be read before they are overwritten. In combination with configuring a special authentication key to access the logs, you can ensure that every operation on the HSM is logged to a remote store before additional operations may be completed.

It does depend on your existing physical security, so that has to be taken into account when designing architectures including it. The micro form factor at least makes it trivial to put into an internal USB port.

And of course, if you require a more enterprise grade tool, you may want to use an HSM in combination with a tool like Hashicorp Vault to manage your keys throughout your orgnaization.

[1] https://developers.yubico.com/YubiHSM2/Product_Overview/

At my prev company we generated keys and split them using `ssss-split` and handed out shards to specific individuals via a keybase exploding message. Our system required at least 3 shards (combined via `ssss-combine`) to reboot.

FYI: Hashicorp vault just uses Shamir's Secret Sharing scheme under the hood: https://github.com/hashicorp/vault/blob/45b9f7c1a53506dc9722...

I highly second the people saying KMS (AWS KMS, Google KMS, or KeyVault).

* The pricing for just storing keys is incredibly cheap.

* At least with Google KMS you can't delete the keys without a 24 hour waiting period (and you can alert on the deletion attempt), so that's a huge safeguard.

* You get key access auditing out of the box.

We use Bitwarden[0] for our secrets. It's open-source with a hosted option. Makes sharing passwords and keys across the team pretty straightforward.

In addition to it, we use envwarden[1], which is a simple open-source wrapper around the Bitwarden CLI to manage our server secrets. It's super simple, but does the job for us well. We can then manage both passwords and keys in one place.

Disclaimer: I created envwarden. I'm not affiliated with Bitwarden in any way however. Just a happy customer.

[0] https://bitwarden.com/

[1] https://github.com/envwarden/envwarden

We commit encryption keys, themselves encrypted, to git alongside the code and everything else. They’re fully versioned and therefore protected against data loss, and we don’t treat dev keys as any different from production keys (just stored in a separate file).

I first wrote about it back in 2017 (1) and we released an open framework for multiple languages/frameworks (2).

1: https://neosmart.net/blog/2017/securestore-a-net-secrets-man...

2: https://neosmart.net/blog/2020/securestore-open-secrets-form...

We use shh for secrets (https://egt.run/shh).

It's designed to integrate really well with your existing CLI tools like vim, xargs, and diff. It offers user-based permissions, and secrets are encrypted into a single file that's safe to commit into your git repo. We can stream secrets out of it directly to our remote servers during deploys.

Unlike Vault you don't need to manage infra to run it -- it's just a file. Unlike cloud secret managers, there's no lock-in.

We use Bitwarden https://bitwarden.com/#organizations to store passwords as well as encryption secrets as a backup.

Application repos have the encrypted secrets (meta) stored in their repos using Ansible-vault and the .vault_keys are stored in Bitwarden.

We store all keys that do not required automated access on Yubikey with the option that requires a physical touch per use.

Usage includes SSH authentication, file encryption (backups and exchanges), git commit signatures and password/secret storage using `pass`.

Copies of the offline master keys keys are stored on flash in safes onsite and offsite in bank vaults, and sub-keys are valid for one year.

We use Hashicorp's Vault for secrets that require automated access.

Disclosure: Founder

https://dev.ionic.com

Utilized globally by individual developers, large enterprises such as JP Morgan & Chase[1], and integrated into the KMS services such as Google Cloud[2].

1. https://venturebeat.com/2019/02/27/ionic-security-raises-40-...

2. https://cloud.google.com/blog/products/identity-security/clo...

At one job I was afraid of losing a set of keys, so as an extra backup, we had a physical copy of the the key printed out as a QR code. This was then physically locked away (safe deposit or similar). But I like this as an option for a “failure resistant” offline backup. The main benefit is that doesn’t assume a stored away USB stick will still be readable while still having a somewhat machine readable format. Yes, the QR code was very large, but could be split into multiple files if needed.

I worked in a place that had a very nice in-house blade server setup with an attached NAS, which was encrypted (at rest, on storage).

This was early in the “cloud” epoch and at any rate they preferred in-house iron for entirely understandable reasons. Also, they needed to do some pretty interesting things, so they had a NAS populated with enterprise-grade SSDs and a native AES-based encryption scheme. This kept the key on what was basically a glorified USB key.

(For those of you who wish to know these details, the network between the blades and between the blades and the NAS was a nicely spec’d fibre channel network, and there was some iSCSI involved. The blades featured Itanium processors, which kind of gives the manufacturer away, and the firm had invested quite heavily in producing very high-performance code for those ill-fated microprocessors, but I digress.)

So... it happened that somebody lost the USB key. Well, not quite. Somebody took it home during a weekend (whilst the system was shut down) and their kid used it for school work.

This proved to be a “significant problem”. There was a backup, and it was encrypted and stored on an adjacent SAN. It wasn’t exactly stale, but it wasn’t entirely pristine either.

There was much woe and gnashing of teeth.

Nobody was fired because the dolt who maintained custody of the only USB key was the founder/CEO, so he couldn’t exactly blame himself.

But, yeah. That happened, sadly.

You need to have a threat model. Then work out a solution from that. Ask yourself, why are you encrypting the hard drivers ?

One threat model might be: Burglars sneaking in during the night and stealing the hard drivers. Then you would store the keys on a different location then the disks.

Then you make it a routine to reboot parts of the fleet, like scheduled simulation/training so that everyone knows what to do when you actually need them.

How paranoid do you want your security to be?

In general I would suggest using a key vault. AWS, GCP, and Azure all have cloud versions that are backed by virtual HSM's built on top of actual physical HSM's. For the vast majority of usages they're good enough. Use admin account management to enforce 2FA/FIDO for all AWS/Azure/GCP logins. (You should be enforcing 2FA with phone/FIDO auth anyway.)

If you need truly paranoid backups, you can back the key up onto a portable hard drive that you lock in a safe in the closet, with a few key people who know the code.

I recommend against using a (cloud-synced) password manager. Cloud key vaults do the same thing but offer specific features relevant to server stuff. And if you want more paranoia, a physical safe is probably safer than extending your attack surface to a cloud-synced password manager.

Also: make sure that you set up a ~quarterly ritual of opening and verifying the backup. For crucial backup fallback systems, you want to make sure you actually use the system so that you know if it fails.

> How does your company manage its encryption keys?

Poorly seems to be the answer industry wide. Both encryption and disaster recovery are hard tasks. Combine them and you have recipe for a total mess.

We use Knox: http://github.com/pinterest/knox

Think of a lighter vault, with ACLs for people and/or machines to access keys and versioning to rotate keys.

In our case, Knox depends upon AWS KMS to "lock/unlock" its storage.

Not sure the "right" way to do it. But this is what we did:

For context: We run a centralised salt-master, salt master unencrypts content using gpg filters as part of variable generation (salt "pillars"). So it's encrypted at rest and encrypted in our git repositories.

What we do/did, is:

* grab a pair of differently branded USB sticks.

* LUKS encrypt the USB sticks; we used a keyfile which is encrypted on our machines with our GPG key.

* encrypt salt's GPG private key with all of our keys.

* encrypt some of the "irreplacable" private keys (IE: CA roots) with all of our GPG keys.

* store it all on the pair of USB drives

* put the USB drives in a real-life vault, give the keys to the office team.

We haven't needed to recover, but it's clearly documented how to recover if anything went wrong.

AWS KMS, and equivalent for other cloud providers. They are in every sense better qualified to handle encryption secrets than my lone ass is.

Lots of decent advise here already...be wary of a complicated (fancy product) approach and security theater....frankly when it comes to secret management the KISS principle should apply, with the caveat that it should be secure of course.

https://aws.amazon.com/kms/

My humble security advice:

1) If you don't need it - don't store it. If you need it - but it needs to be encrypted - probably! don't store it.

2) Think one way hashes with salts, think deletion policies, rotations, small disk drives.

3) You will get owned!!

The dream for me has always been to get LDAP/Kerberos tied into container/VM orchestration, and do everything that way with per-instance accounts.

LDAP is ubiquitous enough as an auth method (how do you auth to Vault? You auth to LDAP with it...) that any service you run or use is likely to speak it.

Why this isn't done more often is a mystery to me and probably the number 1 source of credentials being baked into things accidentally: oh we need a service account into the which uses Active Directory.

Might have answered my own question there though.

I think it really matters how sensitive these keys are.

It's been quite a few years since I interacted with them, but for some keys there is a server somewhere with an HSM installed, and two people have credentials for it. If you need something signed you send it to them, with a justification for why it needs to be signed with the real keys, and they will send you a path to get the signed file, and remind you to delete the signed file when you no longer need it.

This is overkill for some things, and probably would be considered sloppy for others.

Perhaps a dumb question, but how did you restore the data without the keys? Was a prior backup unencrypted?

Put it in a password manager. Nothing can be any simpler than that.

Online HSMs where you can exchange keys. Databases connect to the HSMs for crypto operations + offline HSMs with PCI cards in a safe for root keys.

For other parts HashiCorp Vault/AWS kms

Hi, this issue brought me back in the days when i was just a IT handyman in a small company. The priority of that company was to don't share keys or everything related to them with no one. For no one i mean, third-parties software of wherever a password or an encryption would be watched to someone. At the time, i thought this "obsession" was clearly a sign of mental illness, cause the company was very small and we were in the nowhere of nothing.(maybe nowdays i still think it). Our method was based on a selfmade MD5 encryption script using ruby on rails. Put your password into it, it print it on a datacoin blockchain that generates a univocal MD5 hash. This hash goes around 5 (later 6) local server those collect the encrypted key. Obviously these servers were without an internet connection, running only for internal company purposes.(such as this). For sure a weakness of this procedure was the slowness for obtain a new password or to change it. I think that the most secure place is where there'snt an internet connection. Thanks for bring back memories :D

When managing/deploying code we use SOPS (Secrets OPerationS) https://github.com/mozilla/sops

For standard password style secrets used by Ops, we use Team Password Manager. Which we chose about 5 years ago because it was self-hosted, the database was encrypted, and it had fantastic audit capabilities.

In my previous experience, I was working for an HSM and Data Protection vendor. If you want to be resonable secure to don't lose keys and keep them safe, just use an HSM. If you need to encrypt filesystem, you can use a DP product (most of them are not so expensive) If you want to database content, you can use tokenization services.

I try to push as much to certificate auth as possible. Internal CA keys are stored on an offline USB HSM, which is locked in a cabinet. Access to the key requires 2/10 individuals to be physically present.

There are different CAs for different purposes. There's an intermediate for device management, and another for user or service auth purposes.

A couple of our customers shared this thread with me so I thought I’d chime in. For transparency I am the CEO of Doppler (YC W19), a hosted secrets manager service. I know secrets management isn’t directly related to key management, but it’s a cool security topic we think about often. A one-liner about Doppler - lovable secrets manager built for the everyday developer, that works across all stages, from local development to production, on all stacks, and infras.

For anyone who needs a super simple place to store their encryption keys that works with Heroku and has versioning, I think Doppler could help. It doesn’t have all the fancy (and really cool) features of KMS as it’s designed to be a kv store for secrets, but it could be helpful. We have a free tier for anyone who wants to try it out. https://doppler.com

> I (we) really want to learn what a proper setup would look like

Signing and authentication keys are expendable but encryption keys are worth keeping even after they've been rotated since decryption of existing data may be necessary.

The key can be printed on paper and stored in a physical safe. Paper isn't a high density storage medium but it is remarkably durable and perfect for small amounts of data such as encryption keys. It also counts as an offline backup.

Keys can also be printed as QR codes. They support error correction and enable automatic data restoration. Even 4096 bit RSA keys fit in a binary mode QR code and the smaller ECC keys allow use of high error correction modes, making the data even more durable.

I wrote a binary decoding feature for ZBar in order to support this exact use case:

  zbarcam --raw --oneshot -Sbinary > key

It's available on version 0.23.1 and above.

Anyone using any KMIP based solutions? (looks like Hashicorp's Vault product implements a KMIP server).

http://docs.oasis-open.org/kmip/spec/v1.4/kmip-spec-v1.4.htm...

We keep the encryption key stored in plain text files on client offline machine.

Catch: Client machine is encypted with VeraCrypt. Veracript hidden drives, one password is kept by product owner and another password is kept by head of security.

Offline client machine is key for us.

We rotate encryption keys after quarterly security audit.

We use vault, but sometimes I just `openssl` gpg encrypt the secrets with the keys of all the members of my team and commit the .gpg to git. We all use yubikeys and use them to SSH.

Not ideal, but it works... At least until one of us resign (but turnover is quite low here, so crossed fingers).

We check many of our passwords into a git repository ... but they are all encrypted with unix pass with the public key of each individual in the team authorised to access. Our services require the gpg agent running to launch, allowing them to read the secrets.

Vault for big stuff, git-crypt for file-level encryption of secrets (in Terraform files etc.)

https://github.com/IronCoreLabs/ironhide is a tool we built for managing our own developer secrets. It allows encrypted files to be checked in to git or stored elsewhere. You can think of it similar to gpg, with the upside that ironhide has the ability to change who can decrypt the secret without re-encrypting the data.

Check it out and if you have any questions, feel free to ask here or open an issue in github. We also have a Rust version in the works for those interested in something native.

GitHub Actions, Terraform Cloud, and strict RBAC for everyone on our team so they can't see/change these values on GitHub/Terraform/our cloud environments. This doesn't need to be hard.

The new Google secret manager is a pretty nice way of having an easy to use web interface protected by IAM, with a REST API for applications to pull. You could easily prevent users from deleting keys. It's not as hardcore as Vault but it's a much simpler way of getting keys out of source control IMHO. You can have 2FA and audit logs easily too. Simpler than Google KMS too.

https://cloud.google.com/secret-manager

I've had these variations at work: Checked into the code encrypted, a custom secret vault, stored encrypted in a S3 bucket and downloaded/decrypted at startup.

The main observation I'd make that if you put your keys somewhere only accessible in production, you've made it impossible to test anywhere except production. If you do that, you need to create a process where people can ship some small bit of code to test if the production key setup genuinely works (hint: it won't).

AWS KMS for most thing.

AWS ACM for SSL certs.

AWS SSM for ssh, eliminates the needs for ssh keys.

Not everyone loves AWS (me included), but this stack works nicely in removing the need to ever touch raw encryption key files locally.

I am curious how you managed to restore it if you lost the keys.

Piecing together a key management solution from open source is possible, but if you're trying to encrypt data and are open to commercial products, we do a lot of support of Vormetric.

All of the key management is "built-in" and managed, so there really isn't much overhead. All software-based, with FIPS certified key management. It's very easy to encrypt data this way. It is expensive though.

Disclaimer: while I used to work for Vormetric / Thales, I no longer do.

Hardware key control (HSM) with smart card controlled access.

Locked in a vault.

Keys are an actual secret key (within the HSM), unknown to any human.

Two people with two different access cards have to be present to enable key operations.

So, you don't have to worry about changing keys as employees come and go, because no one knows the actual keys.

There is a whole structure of spare cards stored in offsite secure storage in case a card is damaged, lost, or stolen.

Card set one is stored separately from card set two.

We have a main config/setup git repository which contains keys (and passwords) in git-crypt. 4 people have GPG keys able to decrypt it, and this is used in normal operation for deployment (i.e. for ansible to feed the keys into systems.) The repo is cloned in quite a few places.

None of our secret material is particularly hard to revoke or replace; we don't run an internal CA or anything like that.

the simplest option is to store your keys in aws secrets manager (if you use aws), and then write some tooling around it.

self promotion * You did ask how people do it :), this is my way, Ive written my own service which has been in production for more than 3 years, http://pkhub.io (if you would like to try it send me an email to admin@pkhub.io). This was before aws secrets manager, the tooling is usefull cause I wrote: running your app with its needed secrests dev/stage/prod, accessing dbs, downloading and installing ssh keys to ssh agent, utilities.. end

of course you could write all these yourself with aws secrets manager.

there is hashicorp's vault but tbh it always seemed like way to complicated to setup.

my advice in general would be: to get something secure but simple enough that your engineers can do their work and access the resources they need, without the oh only bob has the keys on his laptop situation.

I have used EJSON and putting everyone's public key in a single keyring. Works but the secrets and keys end up taking up a lot of space.

If you run on kubernetes it has first class support for secrets [0]. You can reference secrets in environment variables, or mount them as files in your containers.

[0] https://kubernetes.io/docs/concepts/configuration/secret/

Shamir's Secret Sharing algorithm [1]. 4 people have shares, with 2 shares required to reconstruct the key.

How the 4 shareholders store their shares is up to them. Mine is in a secure note in 1Password.

[1] https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

We have so many secret values it made sense to build our own internal product, an audited system which holds secrets in a backed up, locked down database. Apps pull from that system at runtime (or deploy time) but they can only access their own secrets using access control. We also use AWS KMS for AWS related resources.

Some on a USB hardware dongle under lock and key. We also run HSMs and the key cards to those require quorum. So the CTO, Founder and head of devops each have multiple cards at home and work such that any two of them can make a quorum with all their cards, or all three can make a quorum with some of their cards.

I put them into a password manager which is backed up both on Dropbox as part of sync and Backblaze for long term.

We save the keys in a cloud vault. For the most important keys I also print them on paper, in text and in a QR-Code (that I generate with an offline tool). It is then placed into a physical safe. This is in case we lose access to our cloud vault, or if the keys are deleted from the cloud vault.

Very badly and inconsistent in my company ! We have a lot of people raising concerns about whatever you want to do. But not many actually have any contributions for how to do it right so every team rolls their own little solution.

I will be very interested in hearing how others do it.

We use Doppler. They're a YC Co. Like an easier to use Hashicorp. Has been great so far.

We use StackExchange's blackbox to check in GPG encrypted secrets into our monorepo: https://github.com/StackExchange/blackbox

Depending on what the keys are for, we use either 1Password or EnvKey for storing them.

Plugging my project Certera as a means to manage keys used for Let's Encrypt certificates: https://docs.certera.io

You can rotate keys and facilitate key pinning scenarios.

Cheers!

They are kept on Keybase >_<

Sadly now forced to figure out what to move to. We're considering 1Password with it's CLI as a short term option, but will be wanting to move to Hashicorp Vault or similar on the mid-term

AWS secrets manager or SSM or KMS for any kind of secrets, keys etc. Works well because our entire stack is on AWS. Otherwise hashicorps Vault would do I guess but that’s yet another service on life support.

Related question: How do big companies (FAANG) store their root private key?

Used Doppler to store encryption keys and other secrets, and it worked well. Was pretty easy to setup and use their secrets store across our dev machines, CI, and production environments.

We use keybase

Nice try, NSA.

My company is too big for me to know about how everyone manages them (my guess would be sharepoint shared folders, with variable degrees of accesses).

My team uses lastpass

We use Azure KeyVault

On clean hardware under four eyes policy:

https://imgur.com/a/FE8sslv

We use Doppler for our encryption keys, great way to easily store dev / staging / production keys in a secure and obfuscated way

We use lastpass for sharing keys internally and we use AWS SSM ParameterStore, this section of AWS is only accessible by a few engineers.

wrote our own kerberos-aaS clone with less features and vulnerable to more internal attacks than plain kerberos and more reliant in a central cert (not cert authority, cert), that is only used sporadically for cross services, not users (there's something else from major vendor there)

and that team now keeps growing and the feature never improves :)

There's a reason gpg supports --armor; print a backup copy of any important keys and stick it in a safe deposit box.

What about https://www.envkey.com/?

Lots of answers, and I still don't see anything valid for a 5 person startup--2 of whom are executive types.

On a USB drive in a locked cabinet. It's not a hardened cabinet. A hard yank would probably open it.

At a prior company they were printed off and stored in a typical file folder.

Generated by an automated process via service now and stored in cyberark

AWS KMS. Access is added invidually using IAM roles. That's it.

anyone use https://www.rcdevs.com/products/spankey/ ?

We have a shared Dropbox account with 2fa hardware and no otp

We email them around lol. Wish we used istio.

We check them into github! (sadly...)

Honeypot post, don't get doxxed

Wouldn't YOU like to know?

We use Azure Key Vault

Lastpass?

Vault.

Notepad

AWS KMS

we use lastpass

I've asked myself this question many times. Or "Who holds the keys to the keys?".

Here is an air-gapped solution:

https://www.arcanus55.com/?trusted55=A55PV2

We use cloud-provider managed encryption because we're not paranoid and don't have legal requirements to manage our own keys.

We don't have SSH keys because it's not the 90's and we don't have servers.