HACKER Q&A
📣 throwaway271723

Team spends all our time fixing CI


I work at a large tech company, I’ve been on the team a few years. We’re building a Kubernetes based service and spend an incredible amount of time on CI related issues. One team member in particular is constantly making tweaks to the pipeline, pushing for improvements and talking about it constantly. The build is broken most of the time. I feel pretty dejected as we don’t seem to deliver much user value. As it’s a big company we just seem to fly under the radar. Anyone else have similar experiences?


  👤 diggs Accepted Answer ✓
I am yet to work on a team with a simple, reliable and repeatable build process.

Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.

I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.


👤 hitsurume
I'm curious to know why you're getting dejected? Is it because you can't get work done and your manager is hounding you, even though its not your fault the builds aren't working? Are you part of DevOps/CI team and are getting blamed for the builds not working? Are you a manager who doesn't know how to deal with a employee that's gone rogue? Do you care too much about productivity as a individual contributor?

👤 thedevindevops
I take it you can't use gated checkins to prevent someone from checking something in that'll break the build?

👤 faangiq
Uhhh try fixing your build? Learn to write proper tests.

👤 jimmyvalmer
CI pipelines are a Good Thing. While we didn't get bogged down by them ten years ago, our testing wasn't nearly as formalized nor robust.

If you're using any one of the popular suites (github, travis, circle), you should be able to fork your own little CI fiefdom insulated from the depredations of the incompetent.

One team member in particular is talking about it constantly.

You have to understand only a small fraction of the tech workforce makes a living off code. The rest of us have to do it by talking.


👤 jimmyvalmer
Rants of twenty-somethings about office politics are fine. Just don't disguise them (weakly) as questions for professional advice.

Then again, nearly all answers on Ask HN are rebuffed by the the poser, so Ask HN is really more akin to craigslist's "rants and raves" or reddit's /r/amianasshole.