Trouble learning things I view as solutions looking for a problem

I've been in cloud / IT space for about 15ish years now, and at some point in the last few years I became quite jaded with all the new shiny tech things. I had a lot of trouble buying into the K8s ecosystem when I could do magical wonderous things with SSH and Ansible, and most conversations with my colleagues at the time were unconvincing at best - they would say K8s can do blah blah, and I would point to our Ansible playbooks that were already doing the same thing. The problem for me was less about the differences in technology but more about finding the willingness to learn something just because I have to.

I've since learned to recontextualize these things in terms of people. CI/CD isn't good because it's better than SSH, it's good because people who speak English can understand a simple devops pipeline but not my custom SSH wizardry. It's a way of inviting developers and even non-tech mgmt folks into my arcane world of development / production and allowing them to see what's going on under the hood. My incentive now isn't about what CI/CD tech is better or worse, rather does it allow my team / peers to understand what I'm trying to achieve and join in. And ultimately that's what I get paid to do - I don't get paid to do cool tech stuff, I get paid to make other people's work easier, or at least that's how I see devops / CI/CD. I know I can always find easier ways to do things, but will they necessarily understand them?

Just my 2 cents. Hope this helps.

one possible way it to realize that it's all re-inventions of the same thing all the way down.

Slack is just IRC with pretty icons.

Reddit is usenet.

Twitter is just AOL AIM away messages.

Etc.

One way that might work is to turn it into a game.

Taking an example from your post, you say: "Why do I need Jenkins and Chef when I can SSH and add scripts and cron jobs."

So work backward and write/post about it.

Starting with SSH, cron, and scripts, here's how you can recreate Jenkins and Chef.

Along the way you'll help other folks who also missed the boat figure out what the new technologies also do as well as give you some fodder for interviews where you can say, I reimplemented from scratch Jenkins using basic technologies and that's how I would use XYZ in this situation.

Or, maybe you could try to focus on jobs where the company is scaling and actually needs to look at whether Chef, Jenkins is the best idea or whether it can be replaced by more optimized technologies that are used to using and have been "battle-tested" to an n-th degree.

Maybe that could help?

Keyword is scale, specifically massive scale. If your scope is just a couple of (web)servers, then, yeah, you can stick with your tried-and-tested tools and processes. But once you start managing 10s or 100s of servers, including a requirement to recover (i.e., redeploy) them ASAP after an outage, then your basic tools and processes might not be enough…

It's your perspective.

The real question is, given all the materials on the internet that can literally tell you why these tools are useful (I mean, surely if you just read the docs you'd find out?!), why have you not done the work?

It feels like you're in your comfort zone and rationalising why you shouldn't do the work. I mean, is finding a job and being employed not motivation enough?

Perhaps look inside yourself and be honest first. It sounds like instead of admitting that you need to do some work and learn some things you'd rather blame outside circumstances for your lack of personal progress.

I don't mean to sound harsh - sometimes we just need a reality check!

Take responsibility, get out there and DO THE WORK. A new world is waiting for you just over the hill.

Many good responses here about why you actually need those CI/CD tools. From my experience I would summarize it as “standardized, repeatable, automated processes at scale”.

I am surprised though that you are getting those questions about XYZs and how would you use them where XYZ are supposedly CI/CD tools. I personally do not understand the hate standardized LeetCode, system design, and STAR interviews get. It is something well-known that you can prepare for, and ace those interviews even if you never had to (or never will) solve a contest-style problem or design a distributed system. These interviews test your general intelligence, memory, tenacity, ability to learn, and communicate. Who cares if I never used ArgoCD and only used Jenkins or rsync or whatever.

I admit sometimes companies need someone with a specialized knowledge to hit the ground running, but these situations are rare. So if your interview consists mostly of questions like “have you used XYZ” and “how would you use XYZ to do ABC” - those are lazy questions, and an imperfect proxy to measure your professional ability. Usually you don’t want to work for companies that ask those, anyway. Anyone smart and with attitude of getting things done can figure those XYZs if and when needed.

You should work on developing a growth mindset. It sounds like you have a typical fixed mindset where you believe that the things you learned and developed early in your career are perfectly fine and there's little benefit to extending yourself further. This mindset will not serve you well professionally and I strongly suggest you work on that as a priority.

Specifically to understand the benefits of CI/CD I recommend reading Jez Humble's book on the topic[0]. He's an excellent writer on the topic. You can also google his name and watch some excellent conference talks from him.

The best way to understand CI/CD is through experiencing an organisation that is building and deploying at scale. But you're between a rock and a hard place if you can't find employment without that experience. Jez Humble's books and talks will help you understand the benefits and capabilities of these systems.

[0] https://www.amazon.com.au/dp/0321601912

I used to have the same issue, before it became my job to understand CI/CD tools. Previously, I was a research student for whom Linux, Git and programming were just hobbies; then my interests got me a foot in the door doing something that I found more rewarding: working on HPC and openstack.

Although I'd heard of CI/CD before, I only had vague ideas about what it really meant. Then I had to learn Jenkins, Gerrit and Puppet to do my new job.

I still don't use these tools for my personal side projects, partly because setting them up and maintaining them requires its own overhead. This overhead is easy to justify when you've got a team who is all sharing the resources and you're managing infrastructure with fleets of dozens or hundreds of servers, but it's less easy to justify as a "one man band."

Conclusions:

* Don't worry about it if these tools don't do much for you right now. Not everything is for everyone.

* It might end up making more sense if you get a job where you need to collaborate with other engineers and people are judging your business based on scalability and site reliability metrics.

When you do get a job that requires such tools, this is why the ones you mentioned are important:

* Chef (or Puppet, etc) are necessary to ensure your dozens of hundreds of servers all remain in sync with a consistent and reproducible configuration. We speak of "cattle not pets" -- in part, that means you don't configure machines individually, instead you enrol them into your configuration management system. The configuration management is stored in Git, so Git becomes your source of truth for how your whole fleet is configured.

* Jenkins: It's a featureful service for running a wide variety of automations without needing to rely on third-parties like GitHub runners. Eg when you are collaborating on code, Jenkins can provide automated reviews, like giving you a +1 if the linting checks pass. When you merge the code, Jenkins can run additional tasks like deployment.

Just don’t learn those things. Sounds like your mind simply won’t latch onto knowledge unless it is associated with an authentic problem. My mind also works that way.

People will tell you to suck it up. Don’t. Find interesting problems and then solve them naturally. You will learn lots and it will feel great. I wrote about this in my book Secrets of a Buccaneer-Scholar.

I have used this method for my entire career since I became a professional coder at 16 after dropping out of high school. It may be important to note that I use coding in my work (software testing consultant, trainer, expert witness) but I am independent and don’t do production coding for commercial products. I don’t have the patience for all that. I burned out as a production coder when I was 19, and have been in testing ever since.

In other words— I get to say screw Jenkins— I write my own frameworks and that’s fine.

Look up these pieces for some of the why’s of the last decade or two:

“Cattle not pets”

“Html,css,javascript for dinosaurs”

“Hypermedia systems”

Kleppmann’s Data Intensive Apps, and a good Postgres book. Data is the big focus today.

When you get asked questions IMO I would just swap out your experience for equivalent, e.g. do you have Jenkins experience - no, but I have created my own automated CICD processes to build test and push. (Part of the reason to do CICD stuff is to have another computer do that stuff in an org with many people contributing, but it is the same principle as you built locally.)

For cloud I would swap out discussing rsync and SFTP in much the same manner, although that is slightly more of a stretch, most people asking those questions either won't know better or will know idiosyncratic cloud stuff can be learned if you are at that level.

In a small tech company with a small audience and finite scale, your 2012 approach will still work in 2024

In any company that operates at web scale, where success means suddenly having millions of users, and thousands of employees, you've got to have a robust way to deploy and orchestrate software.

The key point of the cloud, and technologies like Kubernetes, is that they allow you to treat hardware (servers and network infrastructure) as if they were software. Copy and paste a code template, and you have a data centre of your own to play with. To get more servers and route traffic to them, simply change some configuration. And because you're managing your server fleet via configuration, any changes can be easily automated, permissioned, audited, visualised, rolled back, shared, documented etc.

As a solo developer, you could feasibly create a whole company serving an app or website to hundreds of thousands of users before you had to consider employing any more staff. The cloud, and its infrastructure-as-code platform technology makes that possible.

Startups can focus on their core competency, rather than having to build expensive competancies and making huge and risky upfront capital investments in infrastructure.

> It's not that I actually believe most of these things are "useless". But rather that my mind can't draw a good problem scenario for them. So from my POV they look like solutions looking for a problem. Maybe my experience just has no good frame of reference for those things.

For lots of things your frame of reference should probably be sth like replacing very expensive and very limited engineering time with inexpensive tooling. That could be inferior. But I'm far more eng time limited than I am cash limited. Or, particularly for deploys, (1) blue-green deploys are great; and (2) it's easier to hire eng that don't understand how to use rsync or capistrano than it is to hire those that do. Just like it's easier to use github actions to make it very hard to deploy a branch w/ failing tests than it is to just rely on everyone running tests. (Which, in a perfect world, you could do... but here we are.)

> For instance, why do I need Jenkins and Chef when I can already test, SSH and add scripts and cron jobs no problem.

New members of your team will not be familiar with your scripts and cron jobs, but they might know Jenkins and Chef, learn about them from the docs and fix common issues with Stack Overflow.

Scale, Complexity, Job Specialization, Organizational or Business needs blah blah.

Let me explain.

Imagine you have a team that grows from say 4 to 40 over time. All the initial programmers, business managers etc. have left. One fine day, you get a call from the client saying "The feed has not arrived today, fix it in the next 2 hours or we are going elsewhere (or can you please please please fix it ASAP) ..."

What do you do? How do you prevent this from happening in the first place? What procedures / software do you need to have in place to even know what went wrong etc.

Now let me give examples that these tools help you handle.

1. You need to view all the jobs that run on a daily basis across 400 of your servers. 2. You want to make sure that no changes to anything in production can happen without at least 1 more programmer and 1 more business person signing off on it. 3. A job / script fails partially, how do you assess the impact and move forward (you are not even aware of the existence of this script). 4. You have a backup script / job that needs to run on demand. You want to let your support folks run it or even the business folks, without any programmer intervention (you don't want to get a call every time someone in Japan wants to run this report).

Coming to CI/CD systems, some of their goals are

A. Codify the knowledge into scripts / configs and do not rely in tribal knowledge. B. Ensure that your code is always in a "buildable" state with the new changes (i.e. don't even let bad code in) C. When a build fails, notify the programmers who introduced the bug immediately D. Once a build is complete, send it to other systems for stress testing. ...

Yeah large businesses can have very intricate and complex needs. They rely on generic and configurable software (proven and tested by other large organizations) to meet their operational needs. Also, given that you can hire people that are proficient in these tools, you can focus on the business needs rather then reinventing these tools.

The best way to learn any technology is to use it in a project.

If you're between jobs you have more time to spin up weird experimental side-projects than most people.

It's a bit harder to spin up sideprojects for stuff like Kubernetes because it's pretty expensive to run, but there are plenty of related technologies - like Google Cloud Run or GitHub Actions - that are excellent fodder for small experimental projects to kick the wheels on them.

A great thing about side projects is that they're a risk-free environment for experimenting with new technology.

I think a lot of the mis-match comes from scale. Most companies want to be operating at 10X of whatever they currently are doing. So there is a clear bias towards solutions that can auto scale and are easy for growing companies to hire for.

It might make sense for you to bite the bullet and settle down in some out of the way software shop where there is no conceit of needing to scale. State and Local government come to mind. Be prepared for other tradeoffs and inefficiencies.

The simple answer is, because that is what is being used. The complicated answer is there are multiple ways to do things. And just because you know one way doesn't mean you shouldn't learn another. For uploading files you say you need to use one of three things, until you need to upload a file somewhere that doesn't support those things. You just need to learn some of these technologies, without a frame of reference. Learn a technology to learn a technology.

I suspect this is fairly common.

But I'm also not sure self study will help unless you're willing to lie on the resume. People often explicitly demand experience with the particular tool.

Which is why we see so much "resume driven development". People know they need experience with tools, so they generate it.

That's great it will serve you well.

Once you choose a problem to solve, then you'll have the motivation to learn the tools needed to solve it. So find that problem and solve it, and then maybe you won't need to find a job.

I think some comments are missing the points a little bit. It's not just about scale and I think not at all about not needing new addons to the project not wanting to decipher your bash wizardry. Deciphering Jenkins pipeline scripts or Ruby playbooks for Chef is no easier. In fact, fully figuring out what a Chef recipe will do without running it can be quite inscrutable, way more than figuring out a shell script.

But Jenkins and Chef give you servers. The pipelines or recipes are in a central location with access control that can tie into the same sso or ldap controlling access to everything else in your IT infrastructure. They can run automatically without having to install separate cron jobs on every server you own. Your deployment scripts are probably running from your laptop, which is accessible only to you, not always connected to a network, probably not accessible remotely to anyone, may or may not have any of its own backups, might be unmanaged and impossible for a third party to know you're running it securely and it doesn't have malware.

Organizations are trying to avoid having Dennis Nedry control all of the core infrastructure they rely on.

Sure, as an individual consumer, I might have questioned the utility of something like dropbox, and as it stands, I have never personally used it. But I don't question the utility of distributed filesystems existing when you could just have cron run periodic rsyncs between the root filesystems of some arbitrary number of servers. Yeah, ultimately Chef and Jenkins are just doing what your shell scripts are doing. But Chef and Jenkins are real software. They're well-tested, have a formal development and release process that involves peer review, they're used by thousands of other customers, they come with support contracts and SLAs. Your shell scripts don't have any of that.

The frame of reference to understand these is that they are targeted towards people that are unfamiliar with the preceding technology. The 'new' technology is obfuscated and hyped up in order to differentiate it.

These solutions do not need to look for a problem. They solve very real pain points. I am old enough that I remember doing the things “ssh to server and rsync” way. Heck I do it sometimes still for my personal projects. Here are the examples of the problems that CI/CD tools solve.

1. You rsync your working directory to a server, ssh to it and restart the server. That’s your only server. While you server restarts, your customers experience downtime. If you used modern tools, your tooling would automatically do rolling deployment or blue green of multiple load balanced instances.

2. Your service does not come up because in your local development environment you used the constructs from Python 3.11 that are not understood by Python 3.8 that is pre-installed on the server. What do you do? Upgrade Python of course. This breaks something else running on the same server. Would not be a problem if you deployed a dockerized app.

3. You figured out the Python version problem (all the while your customers could not access your service). Service starts up and crashes after a couple of minutes. Analyzing the stack trace you realized that this is a bug that would be caught by your unit tests - have you not forgot to run them before the deployment. Would not be a problem if running tests was a pre requisite for deployment.

4. At this point you give up and decide to roll back to the previous version. You go through the manual steps of extracting old version, rsync-ing or ftp-inc it again, restarting the servers (good thing you have only one server, right?). All the while service is still down for your customers. Whereas with modern tooling that would be a push of a button.

5. Your boss calls. They are rightfully angry that customers cannot access the service and asks why you did not ask for permission or a sign off for a prod deployment. You said you confirmed with Sue, verbally, but Sue has left the company and any audit trail of that is lost. Modern tools would give you an audit trail of an approval.

6. Jarred by this experience you decide to deploy to at least two servers. Congratulations - now you have to do twice as much ssh-ing and rsync-ing in a carefully orchestrated manner. Let alone that now you have to deploy some sort of a load balancer too. Modern tools do it for you behind the scenes.

7. Suddenly your product is very successful and you get much more of traffic. Your two servers are barely handling the load. And now you cannot do deployments at all because if you stop one of the servers the other will crash under load immediately. You need more servers. Of course you still use physical machines and you need more of them. You call your procurement. They give you two months of the lead time. On the cloud you would just move a slider in some interface and off you go.

I could continue but hopefully you get the idea.

> And I haven't learned how to use them because I can't come up with problem scenarios where they are a possible solution

Well, you have a problem:

> I have no good answer for them because I haven't really learned how to use them

So if getting a job where that skills is important, learn is your goal.