But does it also strike back with these downsides: 1. limits your ability to create "novel" or technically non-trivial systems; 2. moves you closer to the business and thus forces you to deal with many levels of management that influence not only "what" you program, but sometimes also "how" you do it;
Having enough experience with high-level stuff I have this stupid idea that going closer to the OS/metal and wisely choosing tools can somehow protect an engineer from politics and provide an environment with more grunted, real, technical constraints.
Anyone did such a switch? Please, let me know how this corresponds with your experience.
But .. embedded vendors can suck. A lot. Terrible docs, epic levels of errata (e.g., a PowerPC vendor who, at one time, shipped a bigger errata document than the entire manual set for an x86 of the same era), and just.. marginal everything. In some cases, they are really interesting - the Broadcom Fastpath SDK was actually a good mix of excellent and terrible.
But the SDKs can be absolute trash even from name-brand vendors. Without naming names, I'll give an example: SDK contains a linux distribution from an CPU ASIC company shipped as a tar file. You have no indication whatsoever what the reference point for the linux kernel is, no idea what patches it has or does not have, and so on. When you ask them, they don't even understand why you care. Get ready for the fun of massive diffing. The network stacks on the really low-end can be shockingly bad.
From a career standpoint, embedded is a bit of a niche. You will probably always have a job - especially in the IoT era, but even without that. It is typically a somewhat solitary role - you do the code for the XXX block (if drivers) or for some set if functions (IoT-style embedded), but you don't end up doing "broaden your scope" things as much - no system-level nor distributed systems work, which can be somewhat limiting.
Software is, by and large, software. The things that can make or break satisfaction are to me:
- coworkers, clients, and the political environment. This is completely independent of the technology and project requirements. As a consultant I did very different projects with the same co-workers.
- iteration and debuggability. The best for this, for me, is C# in Visual Studio. GDB is an acceptable substitute for the C family. Web applications .. the browser is OK, but now your app is always and forever in two pieces, and you can't cross-stacktrace. Many embedded environments are bad at this; you may find breakpoint debugging and printf debugging are both impossible. Or you have to do an Elon Musk and recreate the crash cause from the smoking debris field.
- fundamentally interesting problem and ownership of work. This can be tricky, but when you get to do things your way and produce great results that can be very satisfying. Again, completely independent of language; depends what impresses you and your circle.
Do you want something you can talk about at parties? I met the last surviving maintainer of the Aston Martin Lagonda's bonkers 1980s CRT-based 6V digital logic dashboard.
I have only a very few times got to choose the language for a new project, or indeed start a new greenfield project at all.
It is more satisfying to work at machine's native speed. Computers are _fast_, especially when you use all CPU cores. ES6 is not bad, but Rust is joy to use. Tooling is solid, and doesn't need periodical `rm -rf node_modules`. It's easier to peel off and debug compile-time abstractions than runtime/VM abstractions.
However, you mention "politics" and freedom how you program. I don't think that has anything to do with whether you write high-level or low-level code. If you work for an organization, you'll have to deal with management, whether you write JS widgets or firmware for a microwave. Programmers bikeshed at all levels, if it's not whether to use CSS-in-JS, then whether to use spinlocks.
The real, technical constraints of bare metal have sparked decades-long microkernel flamewars.
I enjoy the low-level work more, but that's more of a personal preference.
You can for sure solve high-level problems easier, being below it on the abstraction hierarchy, but your work just comes with new problems. I have never beat my head harder against the wall the trying to figure out something that was a chip bug that eventually becomes new errata (if you're lucky enough for the chip vendor to answer your emails in the first placeti get the errata reported).
It doesn't really protect you from politics unless you're the only low-level person in your shop and no one understands what you do, in which case you just stagnate instead. If you're not careful you tend to get paid less too since high-level folk tend to be closer to the flow of money.
I like the work, and wouldn't trade it, but it's not a panacea to engineering org issues in your career.
FPGAs are fun, because you can design logic circuits and hardware, but with some exception you will have to deal with huge proprietary toolchains and closed source 'compilers'. It offers a pretty unique way to solve problems, which is fun to do. Like playing with a tiny but powerful electrical circuit toolbox.
With micro controllers you might have to deal with proprietary compilers, libraries and IDEs but to a lesser amount than with FPGAs. Still fun to get something to blink, move or a display to work.
If you work on micro processors, then you could work on bootloaders, where you more or less concern yourself just to get something working enough to boot another system, or with OS kernels, where you work in a bigger system infrastructure in order to allow as much of the hardware as possible to be available for the user-land by writing drivers. Its challenging because you work on a small gear in the middle of a huge clockwork, that is mostly not written by yourself, and you will struggle to understand.
There are also many special processors you could implement stuff for like DSPs. There you will feel the language limitations but they are very interesting and odd beasts to work with.
Low-Level is mostly about reading hardware and often software documentation (and finding errors in those or the hardware itself) to figure out how to do certain things, and then if something doesn't work, printing out registers to figure out which bit is wrong. Or having to use oscilloscopes or logic analyzers to debug. This certainly isn't everyones cup of tea, but still very rewarding when you actually start to understand how the pieces fit together and things work.
1. Tools are shit sometimes
2. fixing bugs no-one else understands (and so you don't receive proper praise - because manager might not even understand what you did)
3. Literally spending weeks on insane hardware issues, that requires sifting through 1000 page documentation.
4. harder work - less pay (depending on company)
The major positive is it's extremely satisfying when you do fix the things..
That's because there's a lot more choice when you're working at a higher level. If you're working with an embedded system with an AVR, you're choices may be C/C++ and Assembly-- No need to get into an argument over the latest shiny framework.
You'll never really be isolated from office politics or poor management though. Things like feature creep or changing specs etc. also won't go away.
Usually you're working on different types of problems. The technical dangers are different, the type of business logic you're implementing is different, and the types of new things you'd create are different. There's also a huge ecosystem of tools and techniques that you probably haven't even heard about.
If you're just going to make a CRUD app in C or something, odds are you're going to experience the drawbacks you mentioned without much extra benefit. But if you work on actual low-level problems, it can be a completely different world where programming isn't even half the story.
It's been a fun and challenging transition, but my job satisfaction has mostly been determined by the people I work with rather than the subject matter.
Specifically to your points, I haven't really seen any major difference in the ratio between routine and creative programming at the high and at the low level. The amount of politics and business requirements has also been quite similar.
I've been working mostly with Arm Cortex-M, like STM32. Give it a shot - it'll at least broaden your horizons. The Rust toolchain and peripheral access crates make getting started easier than with C toolchains.
It's easy to spin embedded into a multi-domain learning experience. Ie, firmware, PCB design and ordering, enclosure machining, PC-based update tools, part choosing etc.
It feels...liberating: If I think of an electronic device I'd like to exist, I have the tools available to make it happen. Nothing I've done in Python, Javascript, etc provided this.
When you start interacting with hardware, a lot of easy stuff can become more difficult (testing, deployment, debugging, ...) and some difficult stuff becomes easier (number of platforms to support, control over the product, scope of the project, ...). Everything is generally slower when you have to interact with hardware.
Politics is still there as others have talked about, and the complaints about the tools become more "do we really have to code in ANSI C with a proprietary toolchain?".
I'm not saying don't go for it, it's a really nice domain and IMO there's a satisfaction that's unmatched by pure software when you see your hardware project comes to fruition; but don't expect the paradise.
Low-level programming does scratch that itch of "hacking" stuff, although I have had satisfaction with glueing stuff together with Python in interesting ways.
Lots of “that’s a good idea but it’s impossible to predict all the side effects of that change so no.”
You’ll encounter managers who think you should be shipping at the velocity of a high level programmer.
Overall I enjoy it more but the politics don’t go away.
Building tools is what I truly love to do, they support business use cases, and I understand them, and I build my tools accordingly.
For me it's the most fulfilling kind of software development. You need to know your stack to the metal, but you may not need to tinker with it. For example, I do a lot of Java, but high performance Java, I understand CPU cache access times, and roughly what hotspot will generate, what the OS will do, etc. and have the tools to check my assumptions. I don't do the low level stuff unless absolutely necessary, but I use the tools I have to their limit.
You get deep knowledge of software stacks, and get to do experiments, and get to use all the cool algorithms and techniques, and even develop a few new ones.
The part I love most, is that you keep learning. There's so much to learn and it's wicked fun.
I've done high-level stuff, but I end up liking things like rendering engines or layout engines rather than building a dialog box.
I'm ranting, so TLDR: learn your stack, fully, to the transistors if you can, and pick the spot where you have most fun. Money will follow.
in general, everything's exactly the same imo
so if you're interested in the field, go for it
To rephrase the original question: it is not easy to imagine an "Agile" approach to a kernel/network or other kind of systems programming. I would be glad to know there is a chance for a more natural, R&D driven development and bit less fuss about corporate "values".
Thanks guys. Your answers provide some really good and unexpected insights!
I just want to code algorithms and data structures and solve hard domain problems. Not solve difficult programming and operations problems in order to deliver on what is often (at least superficially!) simple domain problems. I think that sweet spot is in games and desktop development.
When you are doing low level stuff it can be liberating because you don’t have to rely on so many abstractions.
I find most of programming dealing with bad abstractions or choosing a library/language that does 90% of what you want but the final 10% is extremely difficult or requires you to abandon some library/language and start from scratch.
The best way to code IMO is to find a way that allows doing low level coding but includes the possibility to jump into the high level whenever needed.
It is frustrating to solve problems with the wrong tool. It is a joy when the tool fits the job. Need to write a web server? Not super fun in C. Need to access raw memory and machine intrinsics directly? Not fun in python.
Unfortunately it's also more political, because the stakes are higher (bigger impact) and for whatever reason more difficult (but talented) personalities seem to get a pass in that world compared to application development work.
Low level programming is way harder with the C++ stack. I blame mostly the language rather then the level. C++ is just a pain in the ass. If you get a low level programming job that uses rust things might be better.
Basically at the higher level, when you hit an error it is (generally) much more trivial to solve... not so with C++. With C++ you can hit crashes and not know where it even came from, this is much more rare in higher level languages.
I know several people who vehemently hate C++ with a passion and if you're going into low level stuff, very likely you'll have to deal with C++.