HACKER Q&A
📣 ebreton

Would you know a “Python idempotent incremental update manager”?


I maintain a fleet of edge devices, including their embedded software written in python. There is a release every two weeks or so, which usually consists on pulling a new set of docker images.

My current pain comes from the fact that, sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.

I am currently writing a (python) script and a documented process for every update, which is gently becoming a pain in the xxx for the units which are not available so often, and for which I have a few updates late.

I would love to have this pile of scripts ran through automatically with a higher level command, and create the skeleton of a new migration script on demand.

Hence the question: how do you manage such remote incremental update manager ? I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.

  👤 ebreton Accepted Answer ✓
My feedback for those who helped me or are interested in following up :)

Pyinfra was the perfect match.

I have used their "operations" to pile up the actions that come with the next release:

- A new package (apt)

- an updtated pip dependency

- checking a file

- and docker-compose actions to update all containers

I was able define my "inventory" easily, which allows me to update my whole fleet with one single command. One caveat here, the timeout is not properly configurable with a ProxyJump. Discussion opened on stackoverflow.

Pyinfra idempotency and dry-run approaches allow me to run the update as many times as I want.

For the next release, would I need to change anything different, I will simply add a new operation. The beauty of it is that all operations are strictly executed in the order I define them, which mean that any devices that would not have been updated with the first release, would be still updated appropriately with the second release.

Cheers !

👤 Someone
You wrote “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”

and “I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.”

I don’t see how having to run scripts follows from the first. Can’t you put the stuff that lives outside docker images in a git repo and pull changes from that?

Initial install would be a git clone and updates a git pull (maybe with a reboot to make sure nothing is running that’s using outdated stuff)

👤 fermigier
Not sure I understand the problem precisely, a couple of thoughts from what I understand:

1) Aren't (Docker/OCI) containers supposed to be immutable (at least, as a best practice) ?

2) Have you looked at Pyinfra and would it help solve your problem ?

👤 yuppie_scum
Dependabot?