Why did back-end development explode in complexity?

Oh you sweet child, you haven't seen complexity until you have deployed J2EE services generated using WSDL, running in an OSGi container, interfaced with EJBs packaged using Ant and deployed into GlassFish, integrated with IIS running on a Windows Server.

Try finding a starter kit for that.

I really don't think this is a productive question or will produce productive answers that aren't tribal and arbitrary.

Some folks will blame "the system" or "the man", others will claim it's not hard and you just need to RTFM, still others will explain how much harder it was "back in their day", and some folks will question the question (like me).

But nobody is going to drop some single piece of information or knowledge that'll "answer" the question; "In 2004 Linus Torvalds declared that all backend systems must contain three additional technologies than the currently mandated two." If only it were that simple.

Things are hard until they aren't, complex until they're simple, convoluted until they're straightforward as can be. The line is different for everyone, so nobody knows what your line is, which means nobody's going to answer the question in a way that will satisfy you, and if they do, it'll only be helpful to a few folks (your folks) and possibly actively harmful to others.

Maybe if you had a specific question that illuminated where you're coming from, folks could provide better answers...

worked at two big tech companies. there is perverse incentive to grow your organization. gets you to director or VP. if you engineer it goes to your promotion packet to get you promoted.

pattern i saw is teams do not build customer feature but work on “efficiency” or “cost saving” project. Ok, so you build something that other teams should use because it will save them time. Sounds very good on paper.

so you do these things to entrench your team DEEP into the tech stack. now people depend on you you use it as reason to get even more head count. ok now the engineers got their promotion but they struggle to support these teams when things not working. So they leave. they got promo, why stay?

managers now use all this ammo to ask for even more hc. it is self serving cycle. Now..development harder. harder to push new features. new people from college come. first they have imposter syndrome and think this is all great and they just not smart to get it. within few year they get it..see processes and tools are idiotic. they learn to play the same game. circle of life continue

The scope of back-end development has grown considerably. In fact, it's precisely "one level of indirection" bigger: "commit is the new save", "push is the new commit", and "repo is the new module". Plus, "full-stack development" now includes containers and orchestration! Which is, of course, insane: distributed systems are, in general, not well understood. Expecting people to understand front-end, back-end, and then just tacking on an academically challenging area is beyond foolish.

I strongly believe that we're now at the limit of human cognitive ability (as evidenced by the tight, and tightening, employment market) and we human programmers are painting ourselves into a complexity corner and we will weep when our employers happily hire the only minds capable of dealing with mechanical complexity, artificial ones.

I think the backend development is still simple it's the deployment that has grown overly complex with Kubernetes/Docker and CI/CD pipelines

Because everyone is prentending that they need to be infinetly scalable like the FAANGs, exacerbated by Cloud providers marketing spiel. In reality nobody knows how to build distributed systems but everybody needs to build them as resume fodder, because they might one day apply for a job at a FAANG... sigh...I sometimes pine back to the days where I saturated the NICs of two quadcore servers with C++ std::unordered_map and FastCGI. Especially the days when I ask what problems the service mesh solves which isn't solvable with internal DNS and loadbalancers in our use case.

BE is simultaneously easier than ever and harder than ever. The "new" complexity in the BE is stuff that has always been there. However, in the past, the infrastructure required so much manual work that it was handled completely independently from the BE.

Now, I can write a BE, configure a Dockerfile, and deploy to a server nearly anywhere in the world without leaving my chair.

I don't think it has. We just have a fashion trend where people love to brag about managing 100 micro-services using Kubernetes. But there's still companies out there where the entire back-end is a single binary on a single beefy server, potentially with a hot spare server running alongside it.

The products we're building are more complex than they used to be, but I don't know if its true that backend development has inherently become more complex than it was in the past.

I've worked on everything from dinosaurs older than me to new greenfield projects using all the latest tech. I think over-complexity emerges in many ways, notably: too much abstraction, ignorance, dogma, and just straight up carelessness.

I think the abstraction problem is the most common and severe though. In enterprise software, I've spent an entire day in the past combing through several layers of services and repositories just to get the full picture of how a response object is being created for a single controller method.

Let me give another example: think about the great debate over ORM vs No-ORM. It's fundamentally a battle about the balances and costs of abstraction in software development. On one hand, you have a (hopefully) simple interface for defining a db's schema, but do I really want to obfuscate the developer from the implementation details of that `findBy` method and trust that the ORM will be 1) used correctly and 2) generate a performant, sensible query? What about security handling? Should I have my developers concerned about preventing potential SQL injections, or just let the ORM take care of that too? The answer depends on a lot of things.

Anyway, I believe for every abstraction:

- something significantly useful should be added (this seems like a truism as I write this, but i've seen a startling amount of useless abstractions)

- the cost should be carefully calculated

- understand where complexity is being added and subtracted

I believe thinking about abstraction in this way would've cut down a lot of the complexity I've seen in softwares past.

To respond to OP's edit: frontend catches more shit because it's largely gotten worse at the same time as getting more complex. This isn't (only) outsiders looking in leveling that criticism—it's people who worked on GUI software in the 90s looking at the state of things and wondering why it's both constantly changing and, in many ways, worse than it was back then. Slower, worse tools producing slower, worse software, while constantly demanding that frontend developers run on a churn-treadmill for—it sure seems—no good reason.

Backend suffers from some similar issues, but not to the same degree. Frontend seems to be subject not just to the effects of the apparently-eternal fat/thin client cycle, as the backend also is, but to some other pressures that lead it to be constantly changing but not really getting better, and even getting worse over time.

Maybe it's a general comment to both back-end and front-end. To me it seems that down in the trenches coding is getting more complex, even though frameworks hide more and more complexity. Languages and frameworks gain features and they don't necessarily shed old ones. I've been working in C# for near 15 years. Each C# and .net release brings more to the table. Often these are features, which simplify things, at the same time the old ways still work. Googling often returns N solutions, each from a different era. I imagine it can be confusing for newcomers to even start. As for the growing complexity in the world of backend, I think, when needed, the best tool against it, is pragmatism.

sarbanes oxley, other reporting requirements. As systems become faster, companies need to save more data. Where I work, we need to be able to reproduce finance reports created years ago using the original data. Doing that makes things far more complex.

When I started out decades ago, systems were far less complex.

Note, the IRS is our best friend. Without them, most of us would be doing something else (if we are very lucky).

Because development jobs pays too damn well, and are too easy to get. Someone famous once said something along the lines of "any idiot can make a thing more complicated, more complex, but it takes the stroke of genius to go in the opposite direction"

Basically, backend dev is so complicated because it's done by fools. When it's not done by fools, it's either not complex or has some very difficult requirements, perhaps sometimes both.

Edit to add, I say this as a foolish backend Dev myself, no offence meant to my own kin, of course!

I think a lot of things actually became easier lately. Deployment? Docker, k8s and cheap virtualization made it much easier than before. Exchanging data and interoperation? Finally people are starting to use comprehensible things like REST and GRPC, instead of trying to invent some byzantine "Enterprise Solution". C/Assembly is no longer the only option if you want performance. Sure, there are more tools around now, but I think the tools themselves are getting somewhat better and more sane.

Backend development? Have you seen what SRE/DevOps people have to deal with?

People literally use hundreds of different tools for no reason at all.

Microservices and services in general where they weren't needed in the first place. Not saying they don't have their use but most companies just ride the hype train and start using technologies for the sake of using them. A couple of years later they end up with an unmanageable spaghetti of tightly coupled services.

Follow up question. Did backend development explode in complexity? What's your source?

I think it's artificial, and the reasons are ego inflation and job security. Nobody wants to admit that all their abstractions and knowledge are made-up and net-negative. But they largely are.

I think complexity is conserved. You can move complexity around in an application but in the end someone has to deal with it.

E.g. the whole microservices vs monoliths debate. While a single microservice is usually relatively simple, you have 10s, 100s or even 1000s of microservices and the entire system design becomes crucial. With a monolith you don't have to worry as much about distributed services and inter-service communication but designing a robust monolith is still a complex process.

Because managing Linux servers is now less complex than orchestrating containers. But people didn’t want to do the former.

The two biggest wrenches that get thrown into a backend system and blow up complexity in my experience:

- 100x in growth, aka scaling.

- Needing to set up reporting data pipelines. Metrics, auditing, 3rd party uses.

Does the frontend need to worry about either of these? Maybe for scaling, they need to do some CDN work, or is that still the backend developer's job?

Best case complexity is not discouraged.

Worst case complexity is incentivized in more ways than this comment has room for.

We live in a society.

It is almost as if complexity is increasing regardless of the software field one is in. Maybe it's due to the fact that engineers exist to build complexity, stacking layers is what we do day to day.

Funny. I am a backend engineer and think so about front end.

Try Frontend development!

i think a couple of reasons:

  - 1. Nodejs
  - 2. Nodejs

"Enterprise" Software development