A place I knew (involved in IoT roughly a decade ago) had written its own SSL certification validation logic (not the crypto part, mind you: just the part that checks “is the date on the cert still valid now?”) for the very first version of its hardware.
It was rolled out in the summer and things went fine. Or so they thought.
On Jan. 1st panic broke out, as the previously undetected off-by-one programming error in the validation logic meant devices could not validate the server’s cert anymore.
There were quite a few layers of learning in that fuck-up, but thankfully it was solved within a few days :)
Somehow, never understood how, I switched to public gists. Well, you can imagine what happened.
The blast radius was null as far as we we know. We got notified very quickly, rotated all the secrets, a few months later we moved everything to terraform and the infra was changing so fast.