HACKER Q&A
📣 OJFord

Do you cache/mirror packages on dev machines with Squid or similar?


I know it's relatively common to cache packages for CI, and AWS et al. have solutions for that, but I'm curious if anyone's caching packages locally, on development machines, to speed up image builds etc. when packages have been pulled before?

A big download (or lots of them) in a single layer that ultimately fails, or the reason the layer itself isn't cached is only one small part of it, is annoying, for example. (I'm here writing while waiting for my fourth attempt..)

If so, what're you using? Squid's all I've found. Are you running it on one machine only or for your whole network on/via the router? Any other tips?


  👤 LinuxBender Accepted Answer ✓
I've seen people use Squid and Apache to cache packages. It takes a bit of tweaking and a lot of ram/disk space but not as much disk space as mirroring the entire repositories.

In my opinion it is far better for CI systems to just cache entire approved repositories and make sure none of the environments can reach the public repositories, ever. Assuming enough disk space this has the advantage of being able to create point-in-time snapshots on all of the repos which is quite handy for validating a point-in-time OS/Application update rather than trying to pin specific package versions, package-by-package. This allows validating a snapshot e.g. 202302091447_UTC and then promoting it to mirrors in a load testing environment, then the load testing team can promote it to a staging environment then the operations or devops teams can promote it to production. This also allows for more time to scan packages for malware, backdoors, etc... e.g. let the red/blue teams have a copy just prior to dev/qa This takes operational dependencies off the public mirrors and reduces network bandwidth costs. There was a point in time where bandwidth was expensive on some cloud providers. Putting snapshots in private VPC's in each region can reduce that cost.


👤 sp332
My ISP has a package mirror. Maybe you can find someone closer on the internet who has one?

Lots of projects have a way to set up your own mirror, but they are all different :(