Where are the places running their setup out of a rack in a rando datacenter grandfathered into an affordable Edgecast plan running a LAMP stack on Debian using borg for backups?
This may read like satire, but I promise I'm seriously asking.
Or am I just way too old?
Related: https://boringtechnology.club/
Just fair warning: you seem to be under the assumption that “boring” means “straightforward and just makes sense.” It does not. If you’re looking for that… good luck! Update if you find somewhere and I’ll join you.
I don't know how old you are, but now that I have running water I think to my old and boring setup I cringe at how much work it was to replace or add to an existing cluster, or to replicate it for testing out new features. How out of date the software was, and how easy it would be for an APT to exploit some known-for-years vulnerability in a transitive software dependency.
I wish I could afford an SRE in my current startup, but the fact that we can run a decently reliable setup that's easy to keep up to date, expand and modify is thanks to all of that web scale technology trickling down to us small fish.
And why not use borg for backups? We still do run Debian (or Ubuntu, but same diff), if it's good it's good.
EDIT: I was considering doing this myself recently, so maybe you'll like the idea, too: Just walk physically to all the mom&dad shops in your area and ask them if they want a free website, if they allow you to put Google text ads on the side. If you have time, you can build those websites and super cheaply host them themselves, and it'll certainly help those shops get their feet wet in the digital age. Plus you build up some ad/link inventory which you can monetize later. The reason why I considered doing it is because I'm 100% sure that people will tell you a lot of technical stuff that they need solved, if you're there and willing to listen. And those small shops are going to need small cloud-free solutions.
Boring, however, is contextual. We have flex hours. We have unlimited PTO (and frequent reminders to take personal time - I already know the data related to offering unlim PTO and how often people then take PTO, thank you very much). We make over $1 million in revenue a year with 6 people and no plans to have exponential growth, coupled with high margins so that we aren't rugpulled by a rainy day (and to take appropriate profits of course). My employees report high levels of job satisfaction and happiness.
Our focus? Maintaining boring legacy software for low-risk clients. We also do some greenfield work, but we like the stability that comes with working on older cash cow software.
The jobs are out there. We're just not super flashy about it.
EDIT: I appreciate everyone who commented :) if this wasn't a burner I'd totally reach out to some of you!
Back in the day when we had to hire a guy to go down to the cage every day to repair yet another faulty machine, we would have given our left arm for the ability to magically spin up a new VM of any size with a baked image by just running an API call. It's freakin' magic.
I also love, love, love plain Fargate. Not as stupidly complicated as K8s, but I can still run containers without ever thinking about managing hosts. Just keep my containers up to date, push new ones, and trigger a deploy. Never have to "upgrade my cluster to keep from using EOL versions" or "migrate ingress to a new API version" or some other ridiculous thing. It just works, and all I have to think about is my app/container.
We did have a brief stint with AWS, but that's thankfully over, partly because their Aurora Postgres-like cloud thing regularly would thrash to death because it doesn't have anywhere to store temporary data so a biggish join means out of memory, partly because the US law that says foreigners have no right to privacy from NSA with judicial review makes it illegal in Europe to use US providers unless data is encrypted client-side (see Schrems II).
I wish someone would start a sysadm contractor company offering setting up and maintaining servers at something like Hetzner. Take a service fee for being ready to respond to emergencies, bill for hours spent, and let Hetzner bill the hardware. Setting up a server isn't that bad, you can do that with a script, but ongoing maintenance and having someone always ready to respond to emergencies (that never occur, almost) is something you wouldn't mind having a competent sysadm take care of, as a developer.
I think there's a huge market for this in Europe with Schrems II, for those that aren't really deeply dependent on the software stack in cloud services. And you can get some beefy machines at Hetzner for pennies compared to Amazon or Google.
The job ads that catch your eye are specifically tailored for those results. They're usually posted from flashy startups who are high on their own supply. I had many an interview with those types of places, and nothing you could say would be good enough for them. That's why those job posting are always open.
What worked for me was happenstance -- I applied to a job that was posted by a recruiter, and once I was in that recruiting company's ecosystem, they started passing opportunities my way. The opportunities were interesting and had "boring" and relatable tech. If I looked up the job ads for those opportunities, the posting itself sucked and lacked the detail I'd expect. I'd have never applied without a recruiter talking to me about it.
There is a massive demand for developers at small and medium sized businesses. Those businesses have no idea how to effectively market their postings, which is why they use recruiters. The startups that know how to effectively market their postings on WeWorkRemotely and such have the applicant pool to be insanely choosy. If you're having trouble landing something, apply to jobs that are close to what you're looking for, but definitely posted by a recruiter. The recruiters will talk to you, and if they like what they hear, the interviews will pour in.
Because the problem domain of banking sucks so much, the tooling is necessarily simple and "unsexy". We like things that fit into a single VM/folder and can be built in 1 click. Anything more complex than this is almost certainly going to fail by default under the combined weight of unnecessary technical complexity and vast (and necessary) business requirements.
It took us until the 3rd year of integration with one of our banking customers before someone in IT even gave enough of a shit to naturally ask the question "so what language/framework/technique did you write
After 12 years of growth we are still happily on:
- 1 Ubuntu server
- 1 MySQL db
- 1 Django framework
- Ansible for deployment
And thats mostly it. Due to Hetzner missing the 27002 certification we moved from metal to Amazon EC2 instances which was pretty uneventful. We also moved to Aurora and the point-in-time restores actually makes me sleep better at night. Relative to Hetzner it's a lot more expensive. Not at all a problem in the absolute sense.Some years back I had a chance to talk with the CTO of a company very similar to ours, and their landscape was 400+ microservices, 50 servers managed by k8s on Azure Cloud, Service bridges, Workflow engines, 5 different DB engines, etc. It sounded very fascinating, but also quite complicated.
Some of my worst decisions over the years has been not having the balls to say no to current best-practices. I've learned the hard way that even though the pressure of using shining-new-thingTM can be immense it's so important to realise that everything always has pros and cons. The most important thing is trying to really understand _my_ specific problem and _then_ using the simplest and most obvious solution that can work.
Obligatory "Programmers are also human" link: https://www.youtube.com/watch?v=Uo3cL4nrGOk
I'm going to say, on a random job board, this is actually the vast minority of jobs. Most, even at FAANG, have lots of unsexy maintenance that just needs to get done. Even in fancy machine learning land (where I work) 99% of the work is fairly 'boring' in the traditional sense.
Also, there is a strong bias to talk about 'sexy' work outside the company. For a variety of dev marketing reasons. Nobody does a conference talk about the uninteresting boring work that gets done.
There’s a ton of business that simply need basic websites and those still need to be developed by someone.
At all of these companies you will be forced to use Internet Explorer 11 and you'll program in a remote desktop Notepad.exe on Windows Server 2012. Doing anything else is explicitly banned. The programming language is a proprietary version of Visual Basic with all the functions renamed and changed slightly so the vendor can control access to the $5k/year documentation, which is still horribly written. You will sign an agreement not to ever ask or answer questions about the language on Stack Overflow too. The vendor of these languages doesn't really exist beyond an email address like "sales@enterprisereadypython.com" - they will gaslight you that this is the real Python and python.org is a scam, "if you're not paying you're the product!". None of your coworkers will know what you're talking about when you mention Git or Linux.
I did startups for 10 years prior. Never ending rat races, constant high pressure deadlines, everything is "ASAP", compensation is iffy (your start ups got to make it).
In contrast, I work for a bank now. Easiest and most well compensated job I have ever had in my life. So easy that I am pursuing additional degrees (which they pay for), and moonlighting my own product offerings, virtually stress free.
They still dabble in some of the tech you mention. But they are much more deliberate about everything. And if they do go with something, it will be around for a long time.
It can be frustrating working in companies that treat IT as a cost center, but there's still a lot of money to be made there. (And as a bonus, the interviews are ridiculously low-pressure compared to Google.)
So unless the company is extremely young (say younger than 10 years) then there will be tech of all ages and all levels of sexy.
We also deploy with Chef and an in-house application deployment system written in Perl, do monitoring with mon, and understand that a sysadmin is a subject matter expert.
Hiring, too.
In my experience only owners who had enough momentum from previous businesses with that setup or who just maintained a fairly small pool of revenue (and so didn't change.)
Also the economy went through a lot of M&A because its hard in the middle, failures are catastrophic, and paying some guy who will probably take you to beer lunches to host your stuff makes you look smart to all the other up and coming CEOs.
For historical reasons, this is also heavily tied to the adoption of EDI. So I'd start looking for boring tech stacks in areas that have EDI documents associated with them: https://en.wikipedia.org/wiki/X12_Document_List
And then you pick local smaller companies and then you have you what you look for.
You don't sound too old but maybe stuck on the past a bit? Nothing boring about a LAMPP stack but there are better ways of doing that, the tool for the job and all.
Nothing makes me want to leave this industry more than having to plan evenings and weekends around on-call shifts I didn't volunteer for and get shuffled around less than a month in advance. My expertise is all distributed systems and database engines =\
What is off putting is all this hyper growth series F funding impending exit nonsense.
An independent and profitable company that wants to sray below 50 or so people sounds like a dream to me.
I work at Home Chef. We got acquired by Kroger in 2018. It's pretty chill. They don't pay what FANG+ does but its a nice tech stack, lower stress environment, and the workforce is really diverse. The Kroger corporate overlords pretty much leave us alone which is great.
Generally mid market companies like this are a little bit more relaxed. Large enough to have some structure and orgs in place while still small enough to do some interesting work without too much bullshit to wade through.
Partly sarcastic rewording: in other words, do you have to track CVEs, updated / broken dependencies, zero-days, encrypt at rest, scale, linguistics, support quick turnaround / Agile, work with people that expect Javascript/Typescript, work with tooling that must run on latest Java or recent C libs, work with managers that think DDD must be followed, etc etc? Sure partly this is just software industry in general, but so many areas of concern balloon when coding for the web.
- Essay: https://www.thenewatlantis.com/publications/shop-class-as-so...
- Book if you dig it! https://www.audible.com/pd/Shop-Class-as-Soulcraft-Audiobook...
I briefly worked for an agency early in my career where most of the clients had WordPress or Magento sites with low to moderate traffic. They would usually only need a few hours a month each, so it wouldn't have made sense for them to hire full time programmers or sysadmins.
I've done both, modern companies make it easy to get things done. Legacy companies are frustrating hellholes.
But bear in mind, there's a big chance you'll be in a vendor lock-in, with little flexibility to work outside of those tools - at least for the larger organizations.
And here, from the title, I was going to advise you to go to Mike Rowe's website, and learn to be a heavy equipment operator, or carpenter, or something.
E.g. years ago a company might have run their own LAMP stack to host a website or a small online store or whatever.
But in this age of SaaS it is easier for these smaller shops to just use something like Wix or Shopify to handle it for them, complete with fancy wysiwyg editors etc.
I used to get requests to "add payments" or something to websites I used to help out with so that they could basically do an online shop. At the time I actively rejected those requests as there were huge headaches around things like PCI compliance and the like. Fast forward a decade or more and with GDPR and Cookie laws etc it is a total non-brainer to use a hosted service where they handle all of this, rather than spin up your own ecommerce solution.
It does not make sense to roll your own unless you have a large engineering team, at which point you move beyond some random thing running in some random data center somewhere.
Of course not everything online is ecommerce or CMS-oriented - there will be niche and custom stuff that is not handled well by SaaS offerings.
1. Banking or insurance. Both industries provide high-paying, great-benefits, low-stress jobs (assuming you're OK with bureaucracy or can ignore it) using anything from ancient technology to modern k8s. Here are some anecdotes:
a. Friend works at a small regional bank with ~150b in assets. Their job is to maintain (amongst other less important ones) a classic ASP application which generates monthly compliance reports that get submitted to the fed. Critical work but trivially easy (if you know classic ASP!). The software runs on a single potato VM; development is done on a clone potato VM. They don't even use version control for this as it has been pending approval for 3 years... For this, they make $150k + 15% 401k match + unlimited PTO (forced 2 weeks per quarter off). Rent in that area is ~800/mo for a 2bdr.
b. College acquaintance does boring ASP.NET WebForms development for a regional insurance carrier. They cover 7 counties out of 30ish in said state. They maintain the member portal for all insured members, which runs in their local datacenter with failover a few states away. Captive audience, known size, no need for fancy scaling or load balancing as they have one active season (open enrollment), easy to prepare for. They plan on migrating from WebForms to .NET Core MVC possibly in the next 5 years. No need to rush, though! Same as above; six figure, good match, lots of PTO, similar rent.
c. Insurance for a major carrier, remote. Think Excellus/Aetna/etc. Boring ASP.NET MVC development with jQuery. The engineer has no idea what hardware they run, as despite working on production features, they never deploy any actual code - that happens 15 management levels up the chain. Said engineer has heard that, if they want a test environment, it is requested through IT and provisioned via VMWare, but has no idea if that is what production runs on. Much higher compensation than the previous two.
2. SMB consulting. SMB accounts for what, 95%+ of businesses in the US? Hit up any city anywhere and you will find 50-100 small IT/software consulting companies. There is plenty of the pie to go around if you aren't looking for exit numbers. These companies tackle everything. Custom AS400 solutions? Sure why not. Software written on a version of PHP so low you can count it on your coccyx? Sure, we migrate that. Mission critical software running on Bob's Laptop but Bob retired and nobody has his password and now the entire line is shut down until we can find a fix? Yeah, we can fix that. SMB consulting is really where it's at if you want wonky, boring stuff.
3. Education of the higher (college) variety. I did a stint at my uni for ~5 years and they are the very definition of boring. "Cutting edge" for a uni is paying Oracle a few million bucks for an SIS system that somehow runs slower than their custom built AS400 implementation that Frank from IT wrote himself 25 years ago. If you're somewhere modern, you might have some sort of VM infrastructure somewhere, but you'll be caught up in budgeting for at least a year trying to figure out what code to bill that shit to. In that time you'll find another department tossing out an old Itanium with 16GB of RAM that will absolutely just barely run your critical Spring 1.0 application tossed together by some students who haven't quite got to a web engineering course yet.
The biggest way to find these is networking. I have not had a proper interview in 25 years - every single job has been a referral over a coffee or beer that starts with, "So I've got this job, it's a little weird, but hear me out..."
On LinkedIn.
This is the best answer that you will get.