I've been on 3-4 interesting projects (about one per year, taking 2-3 months), and the rest has been grind, on-call or ever-shifting "priorities". Tech-wise, I struggled with Kubernetes because we took forever to adopt it, we had the wrong goals, used the wrong tools, lacked in enthusiasm, and then got stuck every step of the way. I learned Go on my own time, just to feel like I'm progressing.
Frustration piled up, and I left, so I could take care of my newborn daughter. I ended up staying home for 2 years, and it was the best decision I ever made.
Now, I'm back at it, doing Go development and Cloud-Native Platform work at a startup. It's been 6 months, and today I screamed at a wall of logs, because some YAML just refused to work, and there are no relevant logs. None.
In the 6 months that I've been here, I achieved nothing. I loathe my lack of progress. The reasons are the same as before: lack of direction, changing priorities, uninteresting work, tools unfit for the job, low-quality solutions, the works. But it's been 6 months! I should be able to create a new cluster on our preferred cloud provider, with tools that were supposed to work. But alas! they don't.
Meanwhile, I've applied for Engineering Management jobs, for a change of scenery; I've held lead roles before, so I thought this would be achievable. I was wrong. I got flat-out rejected for insignificant reasons, or not even invited for interviews. I suspect it has something to do with my 2-year break; people really seem to quickly smile past it, every time I mention it.
Now, I don't know what to do. Is this burn-out? Is this disillusionment? Are these frustrations, ever so real, increasing every day? More complexity, more hype bubbles, more solutions looking for problems to solve. Does it make sense to fret over this? Probably not. So I'll submit this, and close, and go off somewhere for the weekend.
I'm a 25-year-old SWE with 5 YoE and would like to bring in a different perspective here. In my first jobs, I had to deal with legacy systems, most of them being 10 to 12 years old back then (so roughly as old as your career). The tooling in these projects were kept at a minimum: SVN or no version control at all, no infrastructure as code, no containerization – in some projects not even virtualization but just a central development server where you would deploy your source files via FTP.
Day-to-day work was simple and productive because there was no local development environment to set up, no merge conflicts to resolve, no outdated versions to update, no RBAC problems to deal with, no weirdly-behaving containers to kill, just a server and your code. This worked great ...
... until it didn't. 12 years of unversioned configuration changes in production servers, unreproducible development and production environments, unreproducible and unversioned database schemas, black-box database triggers that are in no repository but just on the server, un-atomic deployments leaving a dirty state when the FTP upload fails, black-box DNS configurations, unreliable batch processing via FTP instead of actual APIs, no insights into OS metrics for developers because only the ops department can access them, ops who don't understand what devs need and vice versa, undocumented CI/CD processes, ... The list of deficiencies when your level of tooling is on the other side of the spectrum is just as long, and some of these guys messed up pretty badly.
To me, one core competence of a developer is to find a proper balance. Keeping things as simple as possible, but also as reliable, reproducible and transparent as possible. Believing in innovation, but not falling for every shiny new thing. And always keeping in mind that even if you go for the easy-to-get-started solution, the debt might be due a decade later.
Scale is what reveals the limitions of our 3 inch chimp brains.
Lots of things allow quick scale up these days, but issues scale up too. And our chimp brains dont scale fast enough to solve them. Tools and sytems have gotten sophisticated and complex. But the chimps will stay chimps.
Find small companies doing simple things.
I think there is a confluence of lots of reasons. From what you describe you seem to be working in areas where there are far too many technical options, frameworks, tools, utilities. I think that expectations of what technology can accomplish are way too high and demanding unrealistically fast development cycles.
In my experience, most managements are total morons. Greed, ego, hubris, psychopathy being the operating principles. Absolutely no interest in the workers' welfare, etc. To them developers and IT staff are a fungible resource.
But you have it good. Spare a thought for the IT staff keeping an Oracle, SAP, MS or IBM sourced and Big 5 consulting firm configured system running on 10 year old hardware located in a broom cupboard.