Teaching Greybeard IT

I was just thinking the other day that a non-profit wiki for sysadmins would be invaluable.

It could cover all the basics but also feature case-studies of real-world examples of solving problems, so there could be a sort of knowledge base too that would attract seekers and lead them into the foundational content.

10 or 15 years ago, I remember being able to find detailed problem solving information on Google. Today the search results are a morass of affiliate and SEO content multiplied by plagiarism served from a mountainous dung heap of user-generated blather.

Scott Simpson over at Lynda.com and now LinkedIn Learning has done a fantastic job of covering much of the entry level stuff over the past decade with some courses on intermediary concepts. It is however behind a pay wall. So yes a non-profit wiki / maybe even online academy for "Greybeard IT" would be very valuable. It seems those coming into the work force today not only lack skills to triage and troubleshoot issues across systems but then have had no exposure to why things are the way they are. From personal experience it took years to collect enough stories and insights from those who came before to get where I am today.

Write your own blog. Start with just plain text files served via lightweight web server of choise just to get over the starting hurdle.

Ask the new hires if they care. They probably realize they need to learn AWS, Terraform, Kubernetes and Leetcode, anything else is secondary.

The guys 5-10 years of experience are likely more interested.

Why fix it?

That sort of knowledge is undervalued. The new hires make more not understanding that stuff.

Why not just take the knowledge with you when you retire, and make no attempt to pass it on?

Why is it always up to a lone determined engineer to pass the knowledge on, and not the VPs job to listen to the recommendations to do so, and provide support and training?

Because they think you love it so much, you won't be able to help yourself.

Break this chain. Give them the suggestion - you provide training, time to make content, introduce it to the new people. Explain if they don't get it, what will happen...

And when they say no, take the knowledge and burn it. It doesn't have to be your job to throw yourself on the fire to save their systems. The iron will endure or it won't.

Use the extra time to garden instead.

Please do it yourself! Your level of knowledge is rare and could be helpful to a lot of people. I'd happily buy a copy of "The Greybeard IT Bible" if you wrote/published it as I'm sure many others would. Would even be cool to have a pro version where you do short podcasts/talks on certain tricks of the trade included.

As a retired sysadmin with 30 years experience, I can tell you with a good amount of certainty that nobody will care if they aren't already sysadmins. It is viewed as entirely unimportant to understand these things, and the only thing you need to get a job is good whiteboard/leet coding skills.

The world changed.

I’ve had similar thoughts. I wrote a lot of docs on an internal wiki at a prior company, and also wrote some public docs on my site. In the end nobody really read it and it was too much effort to keep it up to date. That said, I would love to learn from you even though I’m a PM now. I have an insatiable curiosity about how things /really/ work that served me well for more than a decade as an SRE/DevOps/SysAdmin.

What brought this up was a constant and unending battle with our IT department over the use of our Endpoint security tool. It indexes the local disk so that you can query for file hashes across the enterprise at scale.

That means you want to be really careful on what parts of the OS you whitelist. A print queue folder has NO value being indexed and can cause performance issues. And this product persists in having a reputation for being the culprit, even though it hasn't been for more than 5 years.

What we're encountering though, is that the IT staff have NO bottleneck troubleshooting skills. They can't tell you a system is running slow because it's CPU, RAM, or I/O bound...they don't know that the underlying compute could be an issue, they don't know to check /var/log for hints that something might be wrong...they just point, rigidly, at the EDR saying 'it occasionally pops to the top of the process list, therefore, it's the reason the system is slow.'

I hear you. I have the same feelings on "web developers" these days. A lot of them start with some framework and cannot write a simple Form submission in HTML. Many cannot explain the difference between a regular Form submission and an AJAX submission but they surely know how to do ajax.

At a previous employer, our software eng org migrated from on prem infrastructure supported by a systems engineering team to infrastructure in the cloud owned by the dev team.

One challenge was that, while we were lucky to score some talented systems engineers who joined dev teams, it wasn't easy to communicate to leadership that owning and managing infrastructure from the public internet all the way into the database requires knowledge and experience that software engineers might not have.

I tried putting together a PluralSight channel for us to use internally to help onboard new team members. We had access to AWS sandbox accounts where we had no proprietary data, and could build infra as needed to learn and/or experiment with ideas/designs/new services.

You can target people with this experience when hiring, but just getting qualified devs was hard a year ago, let alone people who also had systems experience/knowledge. A lot of the candidates worked somewhere with a QA team who tested their code, a dev ops team who managed a CI/CD process and managed releasing their code, a database team who owned the database cluster, a networking team who owned networking infrastructure, etc.

Really what you're looking for is somebody who likes to code, but also is interested in systems enough to at some point go read the RFC, run Wireshark, etc. I know a little about networking because I ran my own firewall on OpenBSD years ago, have been hosting e-mail and web for over 20yrs, etc. We got lucky with a few hires who lived and breathed code and systems engineering, but it will be challenging to retain people like that in a hot market.

At the end of it all, I couldn't really come up with a way to effectively accelerate learning these skills and getting that experience.

I started the same day as a new grad, and will never forget when, at the end of the day, he said "you mean I have to keep learning?" Yes, yes we do. Still do.

edit: I'd be interested in working on putting content together if people think it would be valuable.

It's not quite aimed at IT/System admin, but Handmade Hero[0] is an excellent resource for learning how a computer works from the ground up. It starts with an excellent intro to C on Windows[1] and works its way up from there. It's really long and involved, but I've found that internalizing the general ethos is quite valuable, even if you just watch a handful of episodes.

[0] https://handmadehero.org/ [1] https://www.youtube.com/watch?v=F3ntGDm6hOs&list=PLEMXAbCVnm...

Maybe it is wrong of me to say it like this, but it seems like there is a trichotomy in the world of working people.

You have people like you, that will voluntarily study their hearts out to find the textbook or proper way to solve things,

You have people that seemingly don’t care, but learn by doing, making mistakes, learning holistically

And then you have the workers that just don’t care and wanna be done with their 9 to 5 at 8 and 4.

You’re going to get a wildly skewed sample looking at HN or Reddit, because, by and large, most people here are the first type.

But as someone that is throughly the second type, I don’t give a flying fuck about VonNeumann and mutexes and race whatever because attaching a word doesn’t do anything to help me understand the problem and the solution. I didn’t spend years reading about theory, I spent years looking at logging software, memorizing errors, and looking at data. I have no doubt that I could look at a server with an issue and eventually figure out the nuance of and solve the issue without ever having heard of any of the proper names.

I promise you I am not trying to be rude or condescending when I type all this, but the honest answer of it is true: some people aren’t academics, and any attempt to give them knowledge in the form of “book smarts” and higher level reading is gonna blow up in your face.

Have you thought of a "retirement" career as a troubleshooting consultant. Hang out your shingle and have two weeks of "fun" every other month and keep the ball rolling.

Incidentally, I have learned much by closing my mouth and listening to those, like yourself, with more experience. So Thanks and keep up the good work.

I think you probably have a lot of valuable knowledge that should be written down.

What knowledge about networking do new hires lack? How does one get comfortable with networking? I'm an undergrad, and my plan for that topic was to learn how to read pcap files and also learn to use wireshark to monitor my network.

I really like the resources reccomended on teachyourselfcs.com

I went ahead and registered greybeardit.org I will create a github pages open source project where people can submit articles, it will be 100% open source.

Probably more examples, less theory. Maybe set up labs for them ex. demonstrate unhealthy processes eating up memory and what tools you can look at to see.

"How Linux Works" from no starch press is very good for a bunch of stuff that is not taught in CS.

Oh man no one cares any more. They fudge through it slowly when there's a problem and forget about it immediately. I've seen teams waste 3 months unable to fix a simple issue and I've walked in and seen it right away. Can I pass these skills on? No because no one gives a fuck. And quite frankly my soul died enough that I don't either. Just over 5000 days until I'm retired.