How “normal” is for a developer to be on-call?

Question

A few years ago, my managers decided that developers should be on call for incidents. We didn't have any sort of first responder or site reliability engineering staff, so almost all incidents, no matter how small, would be handled by developers.Things improved a little bit after some time, but the idea of on-call developers is still very much ingrained in the incident management process.Now, personally, I don't quite understand why a developer would need to be woken up at midnight to diagnose something that cannot be patched by him or her alone.Perhaps the issue is that our process is dysfunctional, but I'm wondering if it is necessary, or just common across the industry.

actionherobiju · Accepted Answer

This is mostly a cost saving technique that companies quietly incorporated as if it is quite normal. Anybody who complained or criticized were branded as not being team players or being old-school. Companies found a way to get rid of operations team and had developers do the same thing without paying them extra. Well, there are some companies like Google who pay for being on-call, but the vast majority does not. So you end up having to cancel your weekend hike when you're on-call, you know, just in case something blows up. You get woken up in the middle of the night to look at an issue which could be in some part of the system you are not familiar with. But who care, this is progress, right? Right? But there is an upside to the devops model. Now developers think through the part after the code has been deployed, they care about the debuggability, about resilience, and logging correctly and all those things which they casually glossed over before. To summarize, in my opinion, being on-call is okay as long as you get paid extra for it, otherwise you're just being used. Have a good day!

ksaj · Answer

It's not normal. It is scope creep.My partner is an executive chef at a private school. It's one of our country's top schools that gets the occasional visit from people with royal titles (and one might soon be King), so think super rich kids: there is a lot of pampering. All of a sudden through this Covid thing we're all working through, there are texts and phone calls at 5:AM "There's a kid with a new allergy coming in today. Make changes in the lunch menu to accommodate them." This is literally a situation where one could ask, "Shouldn't this be an email, CC'd to others involved, too? And maybe even the day before so appropriate food items can be ordered?"It's super annoying and clearly goes against how things are normally done in a well managed business. A lot of companies are clearly taking advantage of all the workarounds this pandemic has forced us to contend with. Except usually the abuse isn't even related to pandemic conditions.A lot of people are technically working 24x7 nowadays.

mark_undoio · Answer

When I've been involved in a "developer on call" arrangement it's been nowhere near as arduous yours sounds - we were supporting users of our software but not an online service, so "help, the central server has gone down!" was less of an issue.
We had an informal rotation in the dev team so there was always a named person who'd make sure any customer issue got an intelligent first response - up until a certain point in the evening. That was to ensure we were able to ask for data we needed from customers in other time zones, then look properly when the team were in the next day.
I have known people on 24-hour call in companies that had infrastructure that was critical to their business (i.e. "as a service" sort of deals).
(personally, where I see developers are already doing some out-of-hours support work voluntarily, I think overformalising would put people off and be counterproductive)

faangiq · Answer

Ah yes the devops scam. Makes sense on paper except that execs don&rsquo;t care about ops or oncall load. So it ends up being yet another way to squeeze the code monkeys.

dyeje · Answer

I would go so far as to say it is a best practice. It aligns incentives. If you know you're going to have to wake up at 2am when your code blows up, you are more likely to make it resilient.