High availability control system software architectures

Question

Hi HN. I'm interested in resources describing the architecture and implementation of critical control system software. I'd like to understand more specifically about how these systems can be or are designed such that loading, patching, and deploying software in these environments can be done with zero downtime.Are there any books, code, or other resources you would recommend?

chermi · Accepted Answer

Can you specify exactly what you mean by "control system" here? Are you talking about software that actually sends signals to hardware that eventually makes to physical equipment that does something?
I'm trying to understand the actual environment. When you say "deployment", what is changed, where does it start, and how far does it propagate?
For example, would one option for zero downtime be to have replicated (2 or more) "control systems" beyond some "layer" (sorry, it's hard to be precise without knowing more) and enforcing synchronicity between those while having only actually controlling at any time. Then, when you are patching or updating, you freeze on one, update the other, then switch to the other? Not advocating a solution, just trying to understand the situation by throwing out an example to talk around.
I'm not an expert in this at all, but if what I'm talking about above is even close to being on track, I'd recommend this book for starters: https://www.amazon.com/Introduction-Embedded-Systems-Cyber-P...

karmakaze · Answer

Loading, patching, and deploying without downtime is not very complicated on the surface of it. Basically almost all cloud product/service providers do this with fault-tolerant network design, routing/load-balancing, distributed/fault-tolerant datastores, blue-green continuous integration/delivery (CI/CD) pipelines.The hard part is being very strict to ensure that every change is safe and/or be able to rapidly/automatically restore a working state to stay within a very low error budget. Each '9' in 99.9.. of uptime is order(s) of magnitude harder.

GianFabien · Answer

Telephony central office switches are an example of the sort of systems you are asking after. A typical switch is designed to run for 30 years with less than 30 minutes downtime in all that time. Ericsson AXE and Western Electric are two such systems I have worked with. More recent example would be the Erlang language and the related OTP environment.

yuppie_scum · Answer

Some topics for you to research:12 Factor ApplicationsSite Reliability EngineeringChaos EngineeringKubernetes