Has anyone here worked on the Windows kernel?

I think I can count the number of kernel changes I've submitted on one hand, but I work on core virtualization that involves a lot of pretending to be hardware and (these days) a lot of poking directly at hardware registers.

I would say James Mickens sums things up nicely in "The Night Watch[0]." For example, you mention debugging with logs and metrics -- this snippet came to mind:

     “Yeah, that sounds bad. Have you checked the log files for
     errors?” I said, “Indeed, I would do that if I hadn’t broken every
     component that a logging system needs to log data. I have a
     network file system, and I have broken the network, and I have
     broken the file system, and my machines crash when I make
     eye contact with them. I HAVE NO TOOLS BECAUSE I’VE
     DESTROYED MY TOOLS WITH MY TOOLS. My only logging
     option is to hire monks to transcribe the subjective experience
     of watching my machines die as I weep tears of blood.”

Mind you, I absolutely _love_ working on low-level stuff, and I wouldn't trade the time I get to spend actually doing that for anything. That said, the complexity of modern operating systems, CPU architectures, interconnects, and peripherals creates opportunities for frustration and confusion that honor no bounds of reasonability or decency.

[0]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf

I worked in the kernel team many moons ago helping improve the performance and then built the .net clr later. Also worked on a NT Filesystem called Cairo. So I guess I have experienced both kernel dev and backend framework dev.

Kernel code is amazing, especially the parts written by the early members like DaveC, MarkZ and others.

For me the biggest part was working with a group of extremely smart people who were very nearly the best programmers in the world.

I really miss that outside of Microsoft. I would imagine that you can get the same experience if you worked with the Linux kernel dev team or some of the other few places in the world like FAMGA where you can work.

My suggestion is to go for it. After leaving MS I found peace by working with the open source community and open source software.

Hope that helps, happy to discuss further.

If MS or any FAANG wants to hire you for a kernel dev position, go for it; they clearly see a lot of potential in you that maybe you don't see yourself. I want to point out a couple of things that you should be prepared for:

> 2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server.

You often can't do this in kernel/OS development work either. Serial printf logging is often required. It can be a real "my tools to debug my tools are broken" slog.

> 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1].

I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

Bad stuff:

1. There will be a lot of infrastructure complexity in the kernel, just prepare yourself for that. Even worse bugs! You'll be fixing a lot of bugs, or looking at a lot of bugs, and most of these bugs are from other teams who are interacting with your component! Just order a copy of Windows Internals and get yourself familiar with how thing work.

2. Old ass engineering systems. Just as the interviewers said you will spend a lot of your time waiting for Windows builds.

Good stuff:

1. Work life balance is amazing actually. Most of the time there's very little pressure for you to get work done, as long as you're doing something no one really bothers you.

2. Since you said you hate designing systems, good news, everything has been designed for you! Your, job will mostly be implementing new features for a component.

3. Windbg is actually great and I will die on this hill. You might have to print some debug logs but you won't be looking at metrics because there's whole teams dedicated to doing that stuff. There's also tools which quickly spin up VMs for you to do some live kernel debugging.

Meh stuff:

1. The pay, could be better. That might change soon though.

I spent about ten years on the Windows Kernel team. In the 90's, so likely very different than what you would experience today, but probably still a lot of the same problems. (Funny to hear people are still complaining about windbg!)

I think you are coming at this from the wrong perspective. Rather than thinking about how to avoid work you DON'T like, think about what you DO like and then decide if the new job would offer more or less of that.

Personally, I've found that every five years I end up sick of working on the same kind of problem and I have to go work on something completely different. Maybe that's where you're at.

Here's my thoughts based on working on Windows in-box code (user mode only, though) across 2 different Windows component teams, totaling about 5 years with a break in the middle. (Split into multiple comments for readability. Some parts removed because other people said it better.)

Most importantly, MAKE SURE YOU KNOW WHAT THE JOB IS! Microsoft people don't try to be dishonest, but there can be misunderstandings between you and your future coworkers about your role, and if you take a job that turns out to be different from what you expected, you will be unhappy.

If you haven't already, you should ask the hiring manager more about what the team does. Try to get enough specifics that you might not know everything the manager refers to, but can easily Google what you don't know: "For example, in Windows 10 version 2004, we shipped the API and implementation for the Windows hypervisor feature that lets third-party VM host software like VirtualBox force their VM guests' virtualized RAM to be paged into the host machine's physical RAM all the time." (Not an actual feature, at least as far as I know.) At this level of detail, you'll be able to judge whether the work is really what you think it is.

Talk to your other interviewers to learn more about the work and the team, if they gave you their contact info or otherwise seemed inclined to hear from you. 3 out of 4 of them are likely going to be your peers, and the 4th is either the hiring manager, another mid-level to senior leader, or a team architect - all will be at least close enough to your team that they won't give you vague generalities.

I've worked adjacent to windows for a few years, and it's not perfect. They have mountains of legacy requirements by the very nature of being windows and other comments about siloed culture can often be true, though I think this is true in most large companies. With that being said, windows kernel developers are by and large extremely skilled and passionate, and working on that team is pretty unique within most of software engineering - rarely do you have the ability to impact so many people.

The concern for me would be the fact that windows is no longer a fully offline experience running on people's desks. A large part of the kernel team is working on features that only benefit Azure and you may get some of the same services exposure there as you claim to be so burnt out on.

Asking to discuss the role more with the team to get a better sense of what they're doing, how they're doing it, and who their customers are seems like the path forward IMO.

Haven't worked on NT kernel, did implement a device driver for it once. Spent first 4 years of my career doing Linux kernel development.

In general, go for it. The level of understanding you obtain by working on the kernel about how things -really- work will make you a better engineer even if it turns out not to be for you.

Things I would caution you about based on the downsides of your current job:

1. Kernels make k8s look simple. Not just simple, childs-play.

2. Debugging experience in kernel-land can vary widely depending on what layer of the stack you are working on and that nature of the bugs. Highly concurrent pieces of any kernel are a nightmare to debug because such bugs are normally race conditions and timings are incredibly sensitive when you get this close to the metal.

3. Whilst you won't have to worry about this initially as you will probably need a few years experience at this level before you design new kernel subsystems I definitely wouldn't consider architecture at this level less complicated than distributed systems. This is because computers these days -are- distributed system. NUMA essentially means you have all the same problems. You do have much more convenient tools for solving them though (at the cost of performance) like HW coherency, etc.

4. Ok yeah, you shouldn't need to worry about this one. You won't be standing up new build systems or anything.

5. Well.. this is the rough part of kernel bugs. When you fk something up you potentially fk over everyone, usually in a very subtle, hard to diagnose and even harder to workaround way.

So yeah, go do it but don't do it because you think you will be getting away from those things because you aren't really. Do it because you think it will be enriching/fun/whatever you want more of in your life.

I used to work in the Windows team before I moved onto a team in Azure. I’ve contributed to both user and kernel mode components. You’ll work with very smart people and they will be very technical. Hiring bar in the Windows kernel team is very high compared to other teams in the company. You learn a lot. Work life balance will also be great.

The other great part is the sheer amount of engineering that will go into every feature you will own. I say “engineering” because it’ll be different from any services work you have done.

Tools that are at your disposal as part of Windows Core team is top notch.

Good luck! You will not regret this.

The design is very different than *nix. The entire ecosystem is different. I think experience with both is eye-opening. There may even be a couple of grizzled old graybeards left who followed DaveC over from DEC. I don't have a clue now.

If you can read and enjoy these books, consider it:

https://docs.microsoft.com/en-us/sysinternals/resources/wind...

The ugh!: You will be a tiny cog in a big machine, spending very little time writing code, mostly banging your head over someone else's bugs. And M$FT still has a certain stink to it these days and that may cling to you.

When I was on the Excel team the kernel devs I talked to knew their stuff and were of the highest quality. More importantly they cared and stuck around to see the consequences of their actions.

The windows org has a reputation of being dysfunctional although I never personally observed this, and personally think even if it were true it would be in the higher level parts of the stack.

Kernel dev was my original dream but I fell in love with Excel during my internship. I definitely think you should go for it.

The Windows Kernel is a quite well-maintained and well-written piece of software. I would say one thing that can be frustrating is that the NT kernel simply does a ton of stuff, and breaking compatibility with all of that stuff is not an option. I once had a one character fix in NLS, that turned into two separate 500+ line AppCompat shims

It leads to code being Complicated, and a lot of Kobayashi Maru situations where there is no good solution to an engineering problem, just a bunch of bad compromises. Depending on your area, you might run into this all the time, or you might not.

I work on Windows (not the kernel) and the best part is that you can work on critical infrastructure without having to be on call. Work life balance is great.

I work at Apple on the kernel team. Congratulations on the job! Getting into kernel development is very difficult.

You said you were looking for things that you might not like about kernel development, so here’s a few:

- Distributed systems. Everything is distributed these days with NUMA, cache coherency, PCI transactions, etc. You’ll have to know how to use the right kind of atomic memory ordering, debug lockless algorithms, and know how the OS scheduler works at a deep level. If this doesn’t excite you, the job might not be a good fit.

- Working with physical hardware. Although in most cases you can get away with a VM, you will almost certainly have to use an actual device for some of your work. The hardware you use might not be 100% functional or even correct- I’ve had to debug kernel issues that ultimately were hardware errata. At Apple it’s easy to get in contact with the silicon design teams, but I’m not sure about Microsoft and Intel/Qualcomm. Working with physical hardware also limits your ability to work remotely easily. Lugging around a bunch of laptops, phones, and tablets in a carry-on suitcase is no fun, especially when you have to take them all out at TSA checkpoints.

- Lack of user-visible impact. Kernel dev is vitally important but it doesn’t get much visibility, except when something goes wrong. New kernel features hardly make the headlines. It can be a little annoying to see your colleagues in app/web dev get recognized for the features they worked on at WWDC or Microsoft Build, while you’re toiling away working on kernel features that very few people care about.

- Debugging. I actually think debugging kernels is not too difficult if you have the right set of tools. At Apple we use lldb and hardware debuggers (JTAG/SWD), and have the ability to take a core dump of the kernel after a panic to analyze later. But since kernel dev is at the core of the operating system, you’ll have to learn to debug other parts of the stack too. For example, you might make a kernel change that breaks the file explorer but only when you visit a specific directory. So you’ll need to know how to debug both the user space process to know what’s going wrong, and the kernel to know why that is happening.

Although there are some negative aspects of the job, there are many more positives. I really like my job and can see myself staying at Apple for many years. I’m constantly learning and working with some of the smartest people I have ever met.

Give the Microsoft job a shot; you can always leave if it’s not a good fit but you will have left a better engineer than when you started.

I work at MSFT on windows kernel drivers - not the kernel directly though. I’d say go for it. Knowing how to program in kernel mode is a lucrative skill set. There aren’t enough people that know how to do it.

In my experience the kernel devs are the best of the best developers. But know that working on the windows kernel will be complicated and require a lot of domain knowledge. Expect to learn about it for _years_ to come.

Part of me fears that you might run into an even worse class of problems as it relates to your points including:

> 1. Infrastructure complexity - Sure, but there would be a LOT of complexity in kernel dev

> 2. Debugging - :) MS kernel is ancient and must be full of cruft. Systems dev is notorious for all sorts of weird timing, coordination bugs.

> 3. Designing systems - You'd have transactions, concurrency, race condition kind of problems which imo tickle the same part of the brain as distributed txns, eventual consistency etc

That said, if you like it, you like it. Can't know without trying. If I were you, and if I had the opportunity - I'd go for a newer variant of this which could look like Apple's M1 team, Tesla's systems teams etc. simply to have a fresh slate to build on.

Kinda seems like if you hate it you could go back to building services really easily. No real downside to giving it a shot! Unless I'm missing another trap door it feels very reversible. :)

You might even be able to return to your old company if you hate it. Tell them directly "hey I need to give this a shot but I'd love to check-in in a year".

Why wouldn’t you take a chance, given how much you dislike what you’re going right now? What have you got to lose?

I did not work on the kernel itself, but wrote NDIS IM kernel drivers (a shim between NIC driver and TCP/IP - like a firewall or load balancer). I found it lots of fun personally. Back in the day I recall using Windbg and later SoftIce for debugging, so when you got stuck there was lots of rebooting to get into a debug session and then careful stepping which could be tedious. It was very challenging, but rewarding and every cycle/instruction counts and every memory allocation had to be carefully allocated from paged or non-paged (never swapped out) pool. I specifically recall you could only call certain functions/syscalls at certain IRQL levels or the kernel would bluescreen, so you always had to know which IRQL level you were being run at. This was 20 years ago so I'm sure much has changed (and probably much hasn't).

As someone who shares your preferences, you should absolutely go for it I would say, even just for the learning experience. Many devs would metaphorically kill to work on an OS kernel team. If it turns out to not be that great after all, you’ll have no difficulty to return to Kubernetes-land ;), or rather more likely, find a different low-level oriented dev job with that Windows kernel experience.

Man I envy you, that sort of thing is the dream. If you don't try, it will always be a what if. The services side of things will always be there if you change your mind.

I'm pretty burned out too with my work (mostly web dev stuff). I've been meaning to get "AWS certified" or learn Kubernetes, but like you, it all seems so crazy to me. I used to love old school "linux administration", but this new wave of tech gives me no interest.

Curious how you get an offer such as this? I've thought about changing job roles, but I really suck at leet coding so I've never really bothered. I figured that as a 40 year old male, no one would hire me for a role unless I was already experienced. Is that not the case?

If you're miserable in your job now then switching seems like there's lots of upside and only a little downside (it could potentially be worse than current).

The main thing I'd consider in this is the work/life balance. Will it allow the same stellar level of that? People vary in their priorities. For some it's interesting/fun problems to solve, for some it's impact, but for me it's work/life balance, presuming of course that the work side isn't a hateful stressfest. No amount of remote work or flex schedule will make up for absolutely hellish work environment, and I had something like that once. (I ended up taking a 30% pay cut that I could just barely afford and got the heck out)

So I'm sorry that I can't speak to the kernel work itself, but instead the framework for making the decision. If the work/life balance will remain constant then you're only risking the possibility that the work won't be any better, and hopefully no worse. If the work/life balance for the new job is also uncertain then (for me) that would be a bigger risk consideration.

Finally, you should consider the worst case scenario: The work is worse, the work/life balance is also worse. How easily can you shift to something better? If you have the chops to work on the Windows kernel then I'd wager that if you hit the worst case scenario you could get out of it without too much trouble, but you're the only one who knows whether or not that's true.

Best of luck to you, sláinte, and may the wind be always at your back.

If you accept the job pay attention to what the Hyper-V team does. They were the best kernel team that I personally worked with and were eager to do things the best way possible.

As much as I loathe MSFT and Windows, I'd jump at the chance to be a low-level C programmer again.

Definitely got for it for no other reason to say that you tried it as you'll always regret it if you don't. The only thing worse than not doing it is leaving such a team for the outside world and realizing that's the biggest mistake of your working life.

i think it's more impressive you were able to land a job in such a vastly different ecosystem. what's your trick against being pigeon-holed?

I have worked on a Windows-compatible kernel. Not at Microsoft, but within the ReactOS Project. And I can affirm that even after 15 years of low-level development, nowadays mostly within various Rust projects, it doesn't get boring. Can't imagine going back to high-level while there is still so much work to do on the underlying foundations..

Congrats on your kernel development offer from Microsoft! I imagine these to be very scarce these days, considering that today's Microsoft seems to be more into services and less about low-level software development.

If you like to chat more about this, just drop me a line. Always interested to hear from low-level devs and aspiring ones. My website/e-mail is in my HN profile.

Just a thought, you might like working at https://oxide.computer/. I'd at least interview there before you sign on at MSFT.

I have worked on it and debugged it in the past. It’s probably some of the cleanest code at Microsoft if I were to guess.

It’s very readable, if a bit idiosyncratic. I would work on it again, given the right opportunity.

I envy you. I would probably pay money to give up my Sr. Manager position at FAANG managing multiple teams doing critical infrastructure work (or rather, trying to) to become an IC embedded developer.

Seriously? You have to ask??

if (offered_nt_kernel_job) accept_right_away()

You can always quit if it doesn't turn out well.

As a former engineer in Windows Core OS team, I can vouch for two topics: kernel code being in great quality and the emphasis Microsoft puts in work and life balance. These two are excellent.

You'll work with people great at their job whom you can learn a lot from.

I also loved having my own window-side private office. I've since heard that Microsoft was trying to adopt open offices, but if it's still there, awesome. Remote work is okay too of course.

Especially in Microsoft, such old historical projects are a huge huge mess. For Microsoft, never join Windows (anything, not just kernel), Power BI, Office (desktop one). Even many parts of Azure that I witnessed when working there, were already a huge and messy legacy. Colleagues, code and development workflow would be your worst nightmare there.

I guess the real issue here is the "work-life balance is stellar". Humans just cant stand being in a comfortable position for a long time. People struggling in their jobs/life will say:"what are you talking about??? I would kill for this opportunity!", but the fact is that we know that growth only happens when there is change and when you are being challenged. This will affect not only your work, but also your relationships. Absolutely any job gets boring after some time, even if you are a Nascar racer. People get depressed when they have "everything". My formula to deal with this is to reinvent yorself many times, and look at your work/life from different angles each time. Understanding that you will be in the exact same position after some time makes you realize that eventually you will have to deal with it.

Sharing because it might give you some Windows history at MS

https://www.amazon.com/Old-New-Thing-Development-Throughout/...

Also read - Showstopper

I'm way less experienced than you are but sort of started out closer to the kernel world (work involves using C mostly) than the webdev world. It's fun if you're interested in that sort of stuff. The things that mostly make you go "ugh" would be testing your work (can't let the kernel slip up, quality is important), the processes involved at some companies in committing something as simple as one line of code, and the lower upper-bar for salary. Maybe Microsoft is a bit wealthier than where I work though.

I once interviewed a guy who worked on various parts of the core windows, naturally being curious I asked a lot of questions about it. I think it would be an interesting experience, but ultimately it seemed very constrained in terms of what you can do, decision making is quite constrained etc. It sounded like you would end up very very focused. Which might be your thing. The guy I interviewed was wanting a lot more freedom. In the embedded world, there's lots of opportunities for low level programming.

To all the Windows folks out there: Thank you for this wonderful OS :)

I know, it gets a lot of trashing all around, the ads, the ui, the whatever - I like the UI and I really like what it offers me for a technical person: ETW traces, Performance monitor, Event fowarding, PowerShell, Windows Terminal, GPO, Windows Defender with advanced features, Resource Monitor, ACLs (Well, auditing file access could have been better :) ), wf.msc, RSAT, HyperV and whatever features under the hood that enables Process Monitor!, ..., ... ,...

> 2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server.

I’m not sure this is really a difference between kernel and backend service development. You can run the server on a dev machine, attach a debugger, and reproduce the bug. If your problem is you can’t reproduce the bug, you may encounter the same issue when working on the kernel. If someone reports a kernel bug, you may not be able to attach a debugger to their computer, so you will need to reproduce it.

I worked on the Windows kernel in the early 2000s. I really enjoyed it - I learned a lot about PC hardware, worked with some very smart people, and although a few projects were ambitious and ambiguous and didn’t really pan out, most of them were really solid; it felt great to be improving something that really made a difference for a huge number of people. I’m a much better developer, with a much better sense for how code serves business, for having spent time there.

Kernel development is not a panacea. I'm a maintainer and you should set you expectations (if you end up upstreaming code) to:

- a lot of bikeshedding

- a mailing list driven workflow

- poor continuous integration

I'm in the same boat. I've been a backend web developer for so long, I've almost become accustomed to the kubernetes churn. I sometimes dabble in kernel dev, and sometimes want to show off my skills to a company like Google and try to go work on Fuchsia or Android or something. But it feels like grass-is-greener thinking, and I'm wary of such things, having burned myself by falling for it before.

I recommend you have a look at some old timer blogs to understand the type of challenges you may encounter.

Raymond Chen is a favourite of mine:

https://devblogs.microsoft.com/oldnewthing/

Microsoft is a friggin dumpster fire, dude. Some teams are great to work for, but a LOT of them are little crews of treacherous rogues who're only looking out for themselves and always ready to stab someone in the back.

You'll run into a lot of the same issues on the Windows kernel that you're getting irritated with in service development land. A lot of infrastructure is already out there, but figuring out how to use it effectively ends up being almost as much work as just rewriting it from scratch. Things are documented but poorly.

For Microsoft specifically you'll often get stuck because of problems with some other team's code, and then find yourself embroiled in multi-week battles with multiple engineers, managers, project managers, and product managers all at each other's throats.

All that said, the Windows kernel itself is a work of art IMO, and kernel development is a lot of fun. I think that if you want to make the big bucks but still work with c/c++, it's getting to the point where your choices are very limited: Microsoft, Google, Facebook, Apple, or Tableau.

Microsoft was terrible to work for, but they aren't as bad as a bad small company. At least there's HR to report gross violations to, they have a TON of great perks, the money is really good, they have good work-life balance, and it's not too hard to change teams (but if you find yourself wanting to jump to a different team, do it well before performance reviews come along, since your manager will definitely give you a bad review and hamstring your ability to move around within the company.. so you'll end up having to change companies. Not the worst thing if you do so, I guess)

EDIT: I just reread your list of complaints.. timing and consistency in the kernel are WAY more complicated than your average web services. There's also a ludicrous amount of red tape around getting anything done. I'm starting to think that you should look at Microsoft as a stepping stone in your career rather than its final destination.. it's still a good place to work for overall compared to random small companies, it'll give a boost to your resume, but it sounds like what you actually want to work on is game emulators or raspi home automation gizmos in your free time.

What level (title is fine)? What team? I can try to help point you at some folks in Windows though it's been a while since I worked in that org. You can reach me via contact info in my profile if you don't mind decloaking.

Perhaps the book Advanced Windows Debugging by Mario Hewardt and Daniel Pravat will get you excited enough to take up "Kernel Development" work.

Definitely do not join Microsoft and definitely do not join any windows related teams. Microsoft is hell. Every team is fully siloed. Culture wise it’s a wasteland and teams as a communication platform is awful

The services world, what does that mean? Kernel development is highly specialized work, so I don't understand where you are coming from and how your previous experience is relevant.

I've managed to just to full stack rails apps for years so I've been avoiding most of those issues. But I feel you on the GDPR, and whatever other regulation is there or coming down the pipeline. It really sucks the fun out of programming. I used to be a systems programmer, and was just thinking the other day that would probably be more fun then building web apps these days.

The answers you will get here at HN will mostly be misplaced hatred against Microsoft, from people who hate MS, because someone else here at HN hates MS.

I think what you should do is, if possible, arrange for a short call with some of the team mates that you are going to work with. Just a casual one about how the kernel team day to day work looks like there. If not with the team, then maybe with your manager.

Unless someone from that team or some ex-kernel team member replies here, all the rest of these posts are just speculative and not worth basing your decision on.