It’s not nearly as bad as you think. If you’re a Let’s Encrypt user, there’s a high likelihood that your servers (and your clients) were negotiating PFS cipher suites.
If they were, a compromise of the private key wouldn’t mean that you could break past intercepted communications.
https://en.m.wikipedia.org/wiki/Forward_secrecy