Therefore, I've been pondering a change of pace. The classic HN answer is of course "create/join a startup", but I've also been looking at areas more adjacent to scientific research.
One option that has come up is the US Department of Energy's national laboratory network[0]. From what I understand, the pay is 33-50% of FAANG, but they do seem to have interesting projects (e.g. the nuclear fusion facility that was recently in the news).
Has anyone here worked at one of them before? What is/was the day-to-day like?
Things don't move fast, as another commenter said. In my area of work, projects tended to last 1-3 years and you'd be on several projects at any given time. In general, it is ICs rather than managers who run the projects. Your manager might say "Bob over in 9876 has a neat project that could use somebody like you, send him an email if you're interested".
You have to acknowledge that the core mission of the DOE National Labs is nuclear weapons. You might not ever come in contact with the mission, but it is there. They have strong HPC programs--because HPC as we know it is basically driven by the need to simulate nuclear weapons. Some people have moral objections to this, and that's fine!
I thought it was a good place to work, all in all.
Edit: I'd like to stress that probably the biggest advantage of the labs is the opportunity for self-directed work. If you can convince somebody (external sponsors, internal R&D funding committees) to give you money, you can work on just about anything. If you can't get funding of your own, you are still more or less able to choose what you work on.
Your work environment will depend highly on which group you're in. Some groups look like a university department without the students: you work in the unclassified area, you publish papers, you can even open-source software (with some effort). Other people spend their whole day in a windowless SCIF working on very sensitive stuff which they can never, ever discuss outside of a SCIF -- but while their public visibility is nil, their impact is arguably greater.
You will find a research group within a division to work for. For example, mine was the Computational Earth Science group (since renamed) within the Earth & Environmental Sciences division.
You will be working with a handful (3+) of research scientists as their supporting engineer. On some projects, you may be doing machine learning work in Julia. On others, you may be coding a fluid dynamics simulation in FORTRAN or C++. On others, you might be doing data analytics in Python. It's highly, highly variable, depending strongly on the PIs you're working with, and can change as frequently or infrequently as you wish (within reason).
Ultimately, I did the reverse: went from a DOE lab into a FAANG company. My reasons are particular to me, but if you're at all interested in a slower paced, more varied and collaborative environment, you can't do much better than working for the labs.
For context, at LANL, I was making ~$100,000 / yr with 3 YOE (circa 2019). This is in northern New Mexico, with such a low cost of living that this amount of money goes about as far as $150k+ up in the Boston area (where I am now).
Pros: - Pay was excellent, especially for the area - Incredibly beautiful country - Very interesting work - Infinite well of taxpayer dollars for equipment and materials - The best job security one can find - Crippling bureaucracy enforced a remarkably safe work environment
Cons: - Crippling bureaucracy made it difficult to move quickly and hit tight deadlines - Internal politics (intra-lab and inter-lab) often adversely affected decision making and program success - Living in a company town - An inability to remove demonstrably problematic employees - A Q clearance limits certain extracurricular activities
Personal experiences with LANL were all over the place and highly, highly dependent on which group one works with. I was very lucky to get in with a group of wonderful people and immediate management that firewalled most adverse developments from higher up the food chain. This is not a common experience but organizational mobility is relatively free, so you can move to work and groups that are attractive.
Worth noting for those coming from private industry: the national labs are institutions first and foremost, not businesses. Organizationally and operationally they exist in a very different mindset and within very different value systems than FAANG-like orgs. The adjustment can be a bit jarring.
My work at LANL will likely be the most interesting and most fulfilling work I'll have done: every day was an adventure into the unknown. The work/life balance was also excellent. If you're a naturally curious person and have an inclination for basic science I'd recommend taking a look at the labs. If you have specific questions feel free to drop them here!
Overall I agree with almost all the positive things people say in other comments - the national labs have a lot of very smart and kind people, there are interesting things (or at least interesting ideas), and if you're at home at a university, you'll find a lot of kindred spirits.
But I made the opposite move you're considering and quit my national lab job and moved to FAANG. Why? Because I wasn't a scientific superstar with a clear vision, and IMO my field (applied math, computational science, etc.) seemed to asking (by which I mean funding) people to do software engineering without many engineers and at the same time still be academics: scientists and mathematicians who spend a lot of time writing grants and trying to publish papers, etc. This made me feel like a liar, writing the grants, and a hack, writing the code. Not to mention that like all academic-type and "interesting" jobs, you are supposed to be happy with the idea that you're going to expend a lot of your free time and energy, and perhaps not be paid quite as much as you would if you weren't pursuing your (supposed) passion.
Industry is for sure a whole other pile of bullshit, but don't assume that the performance review and promotion stuff is any more draining than the things you will have to do to get funding (you'll likely either be spending a lot of time writing proposals, or you'll be funded by weapons money). Don't make trying to get away from money and politics your reason for moving, though there are plenty of good reasons. Good luck!
My recollection is that the whole experience depended on which group you were in, and mine was fortunately very chill. Smart, friendly people who arrived and left more or less at the same time every day. Lots of matrixing and loaning of people from different orgs -- I had the feeling that if I were making a career there I would wind up slowly drifting around between projects.
The biggest surprise to naive me-in-my-early-20s was that "Department of Energy" is a euphemism for "Department of Nukes." Nuclear stockpile stewardship was a large portion of the activity there, and so a lot of your colleagues will be people who are at least vaguely comfortable with that.
There was a ton of "basic research" too -- some high-energy group had a daily experiment that would deliver a "whomp" of a shockwave around 3:15pm most afternoons, there was a room temp. fusion group, lots of interest in assisted driving cars and unmanned aerial vehicles... you just had to appreciate that all the first applications of all this tech was going to be military.
Also the security clearances.... the joke was that the "L" clearance stood for "Lavatory pass" because in our building until you got one, you needed a line-of-sight escort at all times, even in the bathroom. Even for the "L" the process was quite onerous, and I understood that the 'Q' clearance held by nearly all full-time staff was even more burdensome. I heard stories of people waiting for their clearance getting stuck in rooms with nothing to work on. One person in my group basically got sent offsite to some "think tank" or something for several months while he waited for his clearance - I only met him once the whole summer, at a conference.
I interned twice with MIT Lincoln Labs, which among other things, helped build and deploy Radar for WWII which turned into building/managing the technology for Air-Traffic Control, and then turned towards space.
They are primarily a DOD-associated research lab (even located on an US Air Force Base), and so most of the projects have some military-oriented mission. Their mission is entrepreneurial-minded (which I found cool), in that they do the "basic research" and prototyping to prove viability and then the DOD turns over the project to a contractor to make feasible.
While I was there I worked in their GeoIntelligence and Natural Language groups, doing research which I'd ultimately come to understand as being relevant for Project Maven (year 1) and PRISM (year 2). While I'm sure as an intern my contributions weren't directly related to or otherwise leveraged for these programs, in hindsight it was clear that this was the bigger picture that the work was contributing to. Take from this what you will.
Most of the anecdotes that I've read through in the comments mirrors my experience. However, one thing I see missing was how opportunity was "metered" out. Each group I was in was organized like a research lab and the level of your academic progression limited (or opened) your ability to get access to specific projects/work. Their pay scale was also dictated based on this as well. So if you have a BS, your ability to "move up", doesn't exist, but it does if you have PhD.
Ultimately, I was given an offer to work there, but ended up taking a SWE position in the Bay Area because I wasn't interested in continuing my education and felt like my ability to have a career progression at MITLL would have necessitated that.
The pay isn't competitive with the giant Silicon Valley companies, and computing tends to be a little less bleeding-edge than the other scientific domains. The only place computing is at that bleeding edge is in the HPC world since the labs typically have machines like nobody else, so there is a lot of research to do in terms to utilizing them well and programming them.
The only other complaint I see for the tri-labs (SNL/LANL/LLNL) is that pretty much everyone is expected to hold a Q clearance (roughly equivalent to DoD TS + CNWDI). That can be an obstacle for some people. Not a really difficult process - lots of paperwork, interviews, patience while it goes through the system, and then the periodic renewal process and occasional random drug tests.
I personally love working at the labs and plan to stay for the rest of my career. I don't optimize my career around maximization of take-home $. For me, I want fair pay doing something I really feel like I get excited about in an environment where my employer treats me pretty well. The labs give me that.
The labs are not for everyone, but it's the perfect job for some. If you want to work with fantastically smart people and don't mind following a lot of arbitrary rules, it can be a lot of fun. Most of my coworkers intended to spend their entire careers there.
Just like anywhere else, a lot of the day-to-day experience depends on the group you work with. In general, it's somewhere between a university campus and a defense contractor, and the mix is different for each project. The good part is that once you get a security clearance and make some friends in other groups, you can move around.
There might be some culture shock. Most employees have to be US citizens, so the labs are probably less diverse places than you might be used to. And you will really be hitting the brakes while you wait for a security clearance.
I'd say look at the job postings and give it a try! It didn't end up being for me, I don't regret the time I spent at the labs. And it's tough to beat the work-life balance. You can't take a lot of the work home, and most people take every other Friday off (9/80 schedule).
But do consider the location carefully. For example, Sandia and Los Alamos are both huge and have a huge variety of projects, but you're stuck in Albuquerque or Los Alamos which can be limiting unless you really enjoy hiking.
Sometimes prior clearance can be negotiated and a well qualified candidate who is likely to clear might be accepted and placed in a holding pen until they clear but I'm not sure what the backlog is now or even if they do that anymore. At the very least you will almost certainly need to be a US citizen proper.
I liked the work and really enjoyed getting to be a consultant on many projects. Turnover is massive among the researchers because there are few permanent positions, and most groups are heavy on postdocs since graduate students tended to be primarily on campus (UC Berkeley).
If pay is a concern, look closely for the open databases of salaries. At LBNL there is the "book of tears" at the library under the cafeteria, listing every employee and their salary. The exact amount you get varies wildly with the department: prior to unionization in 2016, the range was from 20k to 125k annual salary for postdocs. I hear they raised the floor to NIH levels at least, but I assume they did not make NERSC take a paycut.
The people I worked with were super smart in their fields, but were pretty bad at writing code / handling data outside of Excel so they usually hired interns to help with code-related stuff. Some of the divisions had full-time software developer teams, but I was the only software developer in mine.
The pace was extremely relaxed, deadlines were not tight at all.
I worked remotely, but came in to the office a few times a month. The campus is beautiful, as it is right inside a nature preserve. Everyone there is doing scientific work, so it feels like a real scientific think tank atmosphere and I loved it.
On the plus side: At Fairchild I worked with a technology that made the fastest circuits at the time. I designed some great circuits and new technologies. At Portland State I had some great students, and at LANL I had great colleagues.
On the minus side: At Fairchild status = number of subordinates. I had tenure at PSU, but the institution didn't have a use for me. At LANL success = spending money. I ended up supporting code for the nuclear weapons program. The software infrastructure and software practices were out of date. We were not allowed to update things because that would threaten security. The quality of much of the code was appalling, as was the quality of the science and mathematics behind it.
It's a pretty good gig. The working environment is, AFAIK, pretty similar to the DOE labs. The pay is lower than FAANG, but the work life balance is a whole lot better, the research facilities are terrific, and the research money is a lot better than, say, in academia. We tend not to recruit young engineers that well. We tend to have more success recruiting folks who have been in industry and have come to realize that paycheck size is not the only thing to look for in a career.
There are a handful of non-DOE government labs around that are similar. Not all NASA labs, but NASA/Langley and NASA/Ames are research focused. There are also FFRDCs, which are a bit more towards the applications side (hence, a bit closer to industry than academia). JPL, APL, MITRE, etc. They all have different focuses depending on how comfortable you are being close to the military-industrial complex (MITRE) or not (JPL).
Also, as a suggestion completely from left field: the State Department is always looking for people with an engineering background to work in the foreign service. They do a lot of negotiations around and provide assistance in engineering-related fields, but engineers don't tend to view the foreign service as a career. So if you sign up you get to bypass the normal seniority-based foreign service career path, and you get assigned to an American embassy somewhere overseas (usually, as I understand it, in Europe or Asia). They cover your living expenses on top of your salary. If you love to travel, it's a thing to consider.
I interned in high school at Brookhaven National Lab working on a team that analyzed STAR (Solenoidal Tracker at RHIC) data from RHIC (Relativistic Heavy Ion Collider). I didn't contribute all that much as a high school intern but the program director said at the end that he liked the high school program because he wanted to help funnel and bring people back to help build up the labs.
My experience was that everyone there was extremely smart, but all post-doc and top scientists in their field (the team I worked on was looking for Anti-Alpha particles from gold-gold particle collisions that also helped create Quark Gluon Plasma). So I'm not sure their relative need for regular software engineers.
In terms of bureaucracy, you're still working for the government. The scientists all complained about the layers of government bureaucracy but were mostly okay with it. High-tier science moves at a pretty slow pace; coming from a tech background you might not be used to the slow pace around the actual physical construction of some of these devices, let alone the fund-seeking, approvals, testing, runs, and data collection. and 33-50% is a hopeful estimate. Let's say one is a 500k a year senior/staff SWE at FAANG. at a similar level of experience, one's pay would be very lucky to break 150k.
So fascinating science, layers of bureaucracy, slow moving stuff, PhD's in their fields, and reduced pay. Again I was only a high school intern, but I spoke with the scientists about their experiences so take my recollection with massive salt. I walked away from the summer fascinated by the work and I had a love of physics at the time; but I also left (this was 2010 IIRC?) watching the world of tech explode at a massive pace and thought that I didn't like physics enough ( I had spent my junior/senior year of high school doing a capstone project on theoretical physics and having taken a lot of physics classes). When I went to college the next year, I tried a few engineering courses, and switched to CS. I'm glad I made the switch.
* Pay is very comfortable to live on in the area
* The large majority of my teammates are self motivated and driven which keeps me motivated and on my toes
* I get to constantly experiment with new tech, work on prototypes, and pursue work I'm interested in.
* Almost all of my work is software development but it's rarely pure software work. I'm almost always working with other SMEs and helping them develop their ideas into code
If you're a curious, hard working person it (and I imagine other UARCs) are great places to be
- Exceptional work/life balance that you will not find anywhere else.
- I started at 100k fresh out of masters program, at 5 YOE I was 150k. Goes up steadily YoY.
- The labs operate like a variety of small businesses. This is because there are many projects and funding sources from a variety of customers.
- From what I hear some labs are super relaxed. Like Los Alamos. The working environment is unlike any other.
- It is typical for teams to be in the same building but have no idea about each other. Overlap and rework is common, but is improving.
- Performance reviews depends on the lab and whatever review process du jour HR wants. Where I am you are in competition with your peers. Limited bonus money. So you’ll need to go above and beyond your peers to get it.
- day to day is: you work on one or more projects that last anywhere from weeks to years and report to that projects principle investigator. The PI will interface with the customer and get funding.
- for software development we need to go through strict security processes that dictate what libraries and dev tools we can use. We use self hosted versions of popular tools like Mattermost and Gitlab. No cloud. GovCloud is typically too expensive for most customers unless your working on very well funded projects.
- Managers are hands off and mostly there to ensure corporate compliance activities get done. E.g training, timesheet, perf reviews,etc.
- High level of autonomy. So your expected to be knowledgeable in your area, able to learn quickly and able to work with and network with others to deliver results quickly to customers. For example, you might be tasked with implementing an algo a staff scientist came up with in C++ for an ARM board. It doesn’t matter if you haven’t done that before. Your expected to learn C++, get a demo out, and maybe you can pull in some colleagues you previously worked with who are experts to help.
- There is a political structure in place and reputation is important. While it may not be as intense as FAANG, and underperforming is likely not to get you fired, if you consistently underperform folks will remember and your reputation will be permanently ruined. Which will result in not being picked for more desirable projects. And likely shunned. I’ve seen it happen a few times.
The lab was going to be a brief stint on my way to FAANG but will likely turn into my career.
LQCD is kind of funny because it doesn't YET have anything practical to say about nuclear physics, which is what the lab cares about, but it will someday. So I was pretty insulated from all the weapons+complex integration stuff; my work was 'pure research', which is not that common (though it is more common at the postdoc level, which the lab views as a way of recruiting talent). But unless you can find your own funding (usually from a DOE grant), you're working on something that advances the lab's programs. I can't bring myself to work on nuclear weapons, which is why I didn't stay [there's a lot more to LLNL than that, of course, but it's what my field funnels into, broadly speaking].
The computational expertise for HPC is really unparalleled, especially at Livermore (and Oak Ridge, which I've only visited). They're consistently pushing the envelope in terms of high-performance machines which can address scientific questions that require extremely tight coupling between computing resources, rather than a cluster, and they have a lot of experimental architectures and things like that. LLNL publishes a lot of open-source software; if you've used a cluster in a scientific setting you might be most familiar with SLURM or spack.
The day-to-day can be a bit surreal. At the defense labs people with enormous machine guns thoroughly check your badge on the way in. On your walk to the cafeteria you might pass a beach volleyball court that's inside the superblock [an extra-high-security area where they've got plutonium etc.], next to a machine gun turret. Very few employers have teams that regularly win SWAT competitions.
The food was fine. No luxuries like free snacks or anything else I'd seen my tech-company friends enjoy. No dogs allowed. LLNL has a lot of employee organizations for sports, charities, exercise, etc. Transportation around the LLNL site is via sporadic shuttles but more practically there's a bike share, which is just a bunch of bikes you can leave anywhere (on the sidewalk / by a building).
Is it a change of pace? Yes, but not in workload. You can easily work yourself to death if you let yourself. And politics happens everywhere -- usually in the form of missing the good projects. But the promotion frenzy is minimized, there's a sense of greater purpose in all the projects that is impossible to replicate anywhere else, and the technologists run everything. Managers help connect, but they don't typically determine your day to day priorities. In that sense, you can continuously shop around for good projects and teams without any formal change of position.
As others have said, the work was highly self-directed. As for the need for software engineers, it was definitely there according to my brother. The scientists he worked with were capable in their field, but they needed someone capable of translating their models into something that would execute on a computer. I don't know what exactly he worked on, of course, but he's an ML specialist and was pretty interested in CUDA programming around that time, so maybe that's a clue what kind of skills he was applying.
Anyway, maybe something to check out similar to the DoE network.
Working at our startup almost feels like working at the lab (ie we have scientists and are doing hard tech), but we can also move fast and don’t have the bureaucracy. So maybe consider working at a hard tech startup with a heavy science base!
Among other things, NREL has the lab that tests all of the windmills in the US. There is this squirrely little canyon that gets really weird winds. Almost every day, so you don't have to wait months to get gale/storm winds.
There will be a number of safety training videos shown when you start. We made fun of them: "rattlesnakes are not your friends" or "you and your fire extinguisher". There are lots of paths for walking and rattlesnakes like to get warm in the sun. Please don't pet them. I know they're cute (well, I think they're cute), but they are pretty scared of anything big enough to eat them. Even though we made fun of the videos, everyone was a result of someone else getting badly hurt, so they're important.
The biggest threat to the group I worked with was Congress. Due to a hostile R-Congress, the budget for DOE was cut by about 30%. I expect similar things this year from the crowd in this Congressional session. The excuses for the cutbacks at that previous time was hostility towards Energy Star and conservation programs of any kind.
Public transit to/from the NREL campus is available. Denver has a pretty good public transit system.
The campus there was also different than many other national lab campuses in that it's an open campus and doesn't have the military entrances that others have. It felt like the culture was much more laid back than the FAANG and other corporate cultures that OP mentioned, but perhaps more bureaucratic as well. Again, I was an intern, so didn't have much visibility into that aspect. Overall, definitely a positive experience and I could see myself there if things lined up right.
Expect a very different work culture. If academia is cozy to you, you'll fit in fine.
I did workflow management systems, environmental controls in labs, and lightning prediction software.
Go to https://us-rse.org/join/ and fill out the form to get a Slack invite :)
edit: obviously all the people commenting here are really helpful but you'd get a wider range of perspectives there
My experience was that 80% of the folks are lifers, for better or worse. Work-life balance was great because everyone’s got kids and soccer games and traffic to beat home, but it was still very much a “butt in chair” system, not a “get work done” one. People regularly do “10-40” (10-hour days M-Th) or “9-80” (9-hour days M-Th, every other Friday off) schedules.
Compared with startup life and even FAANG, everything moves slowly. It’s a combination of real security, security theater, and the red tape that comes with being funded by taxpayer money. That plus the extra rules from just being near the nuclear stuff (can’t drink at lunch, guests need a background check, relationships with foreign nationals must be disclosed) made it a slog.
The most hilarious example, heard only through stories: there are two internal networks, the normal one everyone uses and the high-security one that has no internet access. They are physically separate—no cables crossed, no SSH tunnels, no bridges, nothing. But you have to transfer data between them sometimes, so you would remotely load your data onto a tape drive, and there was a tech you would ping who would eject the tape, run it over to the other network, and insert it in.
As with anything rote, the process was soon automated with a shell script. The script would take a couple of minutes to execute, and your data would be transferred. Entire workflows were built on this script. And around noon every day the latency would skyrocket, because the tape runner was on lunch.
One other thing, I was absolutely blown away by the number of outrageously brilliant people I got to work with. Really some next level people.
I worked for Peloton, Pivotal, and few other startups and have to say I love working for ESnet. The entire org is primarily composed of software, network, and systems engineers and the mission is incredibly focused on engineering and sciences. I love not pushing for quarterly profit goals (although we do have our quarterly OKRs), the amazing work life balance (I have yet to work more than 50 hours a week and end up putting in 25-35 solid hours of work in), and not have to worry about being put on PIPs or at mercy to economical forces that are affecting the FANGS.
Yes, the pay is nowhere near the FANGs and the engineering stack is a few years behind but you get back sanity back. I've been able to put more focus on getting my life back after letting myself go during the pandemic and I feel so much better health-wise and mentally.
The fact that they would hire me half time (with benefits) should already say something positive about the place. It's a beautiful location and they have fascinating things going on. Sadly, I was laid off after a few years during a RIF, but got hired back six months later as a consultant (for more $$), which set me on quite a good career path for awhile after that.
I think the national labs (and similar scientist/engineer positions at universities) can be a good option compared to industry jobs, if you can find them and your salary requirements aren't too high. You generally don't have to teach, though depending on where you are, they may like to see academic-adjacent measures of progress like publications, talks, etc. My experience involved far less corporate-style BS than I later experienced in industry, but some very big egos. YMMV.
Good luck w/ your change of pace, I hope it goes well for you.
Simons Foundation (offices in NYC and Berkeley) Allens Institute (Seattle)
OpenAI would be more research focused as well
I have no affiliation with any of these nor have I worked at any of them, but I'm also looking towards a career in scientific research.
Also, the national labs model gives you a chance to work on big teams with people from diverse backgrounds, which is a lot of fun. For example, a software engineer may find themselves working day to day with physicists, biologists, etc. and learning enough about these fields over time to make novel contributions to them.
That said, it was hands down the most interesting place I've ever worked. I don't think it would have been nearly as interesting if I were a remote worker though. Half the fun was taking to the "lifers" there. Learning the lore of old experiments and asking them to show you around on lunch breaks.
The pay is good for government work. You can continue to do fundamental science research. You get great benefits, and they do some wicked cool science and engineering there (read about NIF and Z-pinch). I didn’t get to see any of the wild stuff behind security, so I can only imagine.
- PHd professors were basically seen as demi-gods. It was their way of the highway, down to the micromanagement level, and they make terrible managers
- Worked seemed pointless. "Here is some money to look into this, this will likely never get implemented"
- Implicit push for producing positive results rather than saying this won't work.
- Just dreary environment overall.
Also worked at NavAir at Paux River.
- salary was absolute shit for the amount of technical knowledge/work required. I feel like setting up the instrumentation (hardware/software) for an experiment in a wind tunnel is orders of magnitudes harder than writing a web service, but maybe thats just me.
- Got to work on some cool things, but gov culture and nepotism is too much. People with military experience automatically put ahead career wise above those without. No thanks.
>but nowadays I find their performance-review and promotion obsessed cultures to be really draining. Worse, those negative feelings seem to be leaking into my personal life and slowly alienating friends and family.
As someone that has worked for FAANG for 6+ years, this isn't an issue with FAANG but an issue with you. Its fairly easy to move around FAANG (although probably not right now) to find a niche job where you can cruise control on your salary without anyone bothering you too much.
As someone who had all their personal information stolen in the famously known OPM data breach, think twice about if you really want to do the background check with SF-86 for a Secret or TS clearance. It's a real pain in the ass. Even if you have an absolutely squeaky clean record.
https://www.google.com/search?client=firefox-b-d&q=opm+data+...
As somebody else mentioned, FNAL is very open in contrast to the other national labs. The guards are very friendly, and tourists come in to see the buffalo and the architecture.
(One potential downside is that your experience in industry does not count much and it might be hard to get in without a proper academic career. That is not a problem with the national labs, but academia in general.)
I had this letter from the lab founder Robert Wilson on my wall, which reflects the spirit well: https://www.fnal.gov/pub/today/images/images09/WilsonBureauc...
> Dear collegues,
> an all to common failing of large institutions is to fall into the bureaucratic morass - complicated procedures, red-tape and all that. That's terrible.
> Let's try hard to keep the good old can-do informal spirit of Fermilab alive! I ask each of you to be intolerant of creeping bureaucracy
>
> Bob Wilson
It's employees love to stroll off onto online dating sites and list of things like drug use, foreign ties, or... other things that would preclude a clearance, then act aggrieved that someone like this poster thinks people who want to weaken the internet under the guise of national security shouldn't also obstruct them from private employment and have done so since the days of "don't ask don't tell".
I interviewed with CERT a couple times... and I got a PA MMJ card after rather than have one more job interview treated like a free consulting session, and I suspect the solarwinds breach happened because so many people who shouldn't have clearances use their positions to obstruct perfectly good candidates because they feel threatened if the tech savvy at risk youth of Appalachia are lifted out of precarity.
Do not work for these people -- stay in the private sector then get a barista gig or something when you have a nest egg.
Yes, you earn less than the private sector, BUT, you get to work on important stuff that makes a difference (sometimes) and with some very smart people. Plus, you never get bored. Labs like Argonne have plenty (~95%) of open science stuff, so you don't need a clearance.
Finding a job there is a challenge though and luck plays a large part.
The only drawbacks, besides the pay and location (for other remote labs) is the funding. If you want to pursue your own research agenda, you need to find funding, which can be challenging. The lab overhead rates are pretty steep, so it gets annoying. If you don't want to be a PI, you will need to find a PI who has the funding to support you. How secure that funding is and for how long depends. But if you have a skill set that can be used across fields and domains (e.g., applied stats or data science), then it gets easier to choose PIs and interesting projects.
https://www.techstars.com/accelerators/industries-of-the-fut...
The cafeteria is way less fancy than tech company cafeterias. :-)
As a few posts have pointed out, there are national labs that do only unclassified work (LBNL is one). So you don't have to get a clearance or be prohibited from accessing lots of places or conversations, and you don't have to work on weapons. You do still have to sign a loyalty oath as a state government employee (the lab being managed under contract by the University of California), something that became highly objectionable to me in retrospect.
Life seemed to move slower here, nobody was up in arms about timelines to sell products to new customers for lower advertising costs, and though annual budget processes were mundane, that was for the directors to deal with.
The commute was the worst part. But the people and work made up for it.
(We're hiring by the way; a bit stale but [1] is still relevant.)
I loved every minute of it and, by proxy, got to see some really really interesting projects and meet some really interesting people.
All of the Labs or these DoD Research centers typically require Security Clearances so that means you need to be a US Citizen and have a squeaky clean background and it also precludes things like using Cannabis.
Software engineering is vastly different than the physical sciences (I've worked in both).
I worked a bit with the Sandia and some folks from national labs on the corporate side in the energy industry. The only software engineers I was aware of were ones that did IT work (Integrated workday, salesforce, etc). The scientists and mech/chem/EE engineers I met were the ones doing all the physical sciences work.
Staff levels probably pretty good if your motivation isn't to move upwards and focus on whatever your interests are.
These are my speculations.
I don't even work at a FAANG company but this resonated with me so much.
I worked at Oak Ridge National Laboratory in High Performance Computing and did not work with anything directly nuclear at all. The HPC efforts of the DOE are under the Office of Science (separate and at the same level as the NNSA) which is focused on more wider scientific impact and application than just nuclear. The Office of Science has a number of program offices that focus on all different kinds of science from basic energy sciences/physics to biological/environmental and scientific computing (where HPC is funded in DOE).
I agree that the work/life balance is great and it is definitely a slower pace than what you would find in industry. The lab system is huge and there are plenty of opportunities but on the Office of Science side I like to break it down between what I think of as a research group and a user facility.
Working in a research group is much like academia, they mostly require a PhD and from what I could tell performance is judged on publication output. These folks also write grant proposals that come from DOE program offices for funding their own research. In some cases I have seen these groups employ non-PhDs to be computational scientists and write code.
The user facilities are long-running projects funded by the DOE at the labs to provide specific capabilities to researchers, sometimes just for DOE scientists but a number of them are open to scientific researchers all over the world. This is where I have the most experience where I worked at ORNL's National Center for Computational Sciences on the Oak Ridge Leadership Computing Facility (OLCF). These projects are generally well funded and have all kinds of interesting challenges to solve. For example, the OLCF has consistently deployed the number one supercomputers on the Top500 list and it offers those computational resources to anyone through their allocation program INCITE which supports many different computational modeling and simulation experiments. Other examples of user facilities at ORNL are the Spallation Neutron Source and the High Flux Isotope Reactor.
One thing I have noticed since moving from ORNL to industry is that the sense of shared purpose does not extend as far in the lab system as it does in company. What I mean is that with the small research group and with a user facility like the OLCF there is shared purpose with the people in those groups but it does not go much beyond that. A lab is generally made up of lots of different research groups and a few facilities but beyond the drive for "Science!" there is not a lot of shared purpose or collaboration at a macro level. The analogy I use is that a lab is a bunch of small dinghy boats that are all generally moving in a similar direction but a company is a single ship with a specific purpose driving it forward.
Overall I loved my experience at ORNL, I learned so much working with so many smart people and made friends that I will have for life.
There's also annoyances coming from political things, such as the budget not being done on time so no one gets paid or there are furlough days.
At least where I was at, a sky-high tolerance for bureaucracy was critical to personal happiness. You will work among the brightest people you ever met. But also some colleagues will be 100% checked out. Aspire towards those that inspire, not the working retired.
Their job classification system is such that you will want a Masters degree. I joined with a BS in CS, and until I got my MS, I was categorized as a technician and was paid the same as say someone who soldered and assembled electronics - just over half of what someone with masters in CS would get with simular rankings. I've heard it has improved since then, but there is still stong bias towards those with a masters. For many engineering jobs this makes sense, but it is out-of-touch for computer programmers and security researchers.
Apart from that while the pay is less than SV, all the labs except LLNL are in parts of the country with much lower cost of living as well, so the pay is pretty darn good for the area IMO. Benifits are good, but not exceptional. Work-life balance is great. I've hand a handfull of month-or-two long crunches in a 20 year career where I had to work 60 hours a week. The rest of the time I work my normal 40 hour schedule and go home. They have standard, 9-80, and 4-10 schedules as options (which nearly all managers will approve). After years of having every other Friday off it would be hard for me to go back to a normal schedule.
The actual work varies wildly with the project you are on. Nuclear Weapons work is extremely slow and process heavy as you might imagine, others are more nimble. Projects I've been on have varied from solo development writing software for the engineering next office over, to small agile teams on quarterly releases, to 5-year waterfall development cycles which a huge team. I've done everything from microcontroller software for sensor systems, realtime streaming data processing, desktop data analysis software, web tools for managing data stores, to pure algorithm research. And that is just a small sample of projects going not even touching supercomputer simulation or security work, that I have no experience with. I feel like I have had a great balance of interesting and stimulating technical work and necessary grunt work. Needless to say it is hard give a single "this is what working at the labs is like", and individual experiences will vary.
There are some differences from industry that are independent of the project, particularly around security. It is not uncommon for software development to be done at the unclassified Official Use Only level, but (production and test) data to be at the classified level, which is done on separate networks (or stand-alone computers). Moving between the two environments can be a time sink. Getting approval to use third-party libraries on classified systems can be a very slow process (weeks at a minimum) depending on the network. If the generic security plans won't work for your project developing a custom one can take the better part of a year. There are many security processes for which I completely respect the purpose, but am flabbergasted at the inefficiency of the execution. Contributing patches back to open-source projects is painful enough that it is rarely done. There is some third-party software that is prohibited (like JetBrains due to connections with Russia), and cloud based tools (on the public internet) are obviously not allowed. You need to be constantly mindful of what you type/say to maintain OPSEC and avoid leaking classified onto OUO systems, or leaking OUO to friends and family.
They are allowing WFH now, but most managers for most jobs will want you start on-site to help acclimate to the security culture, and to live in town to be able to come on-site to work in the classified environments when needed. You will need to apply for a security clearance once hired, and some projects are better than others at finding meaningful work for you to do while waiting for the security clearance to be granted.
As far as ethics go, on one hand you won't be asked to write dark-pattern advertising-driven manipulative spyware. On the other, most work will be related to defense applications directly or indirectly. There are some projects related strictly to energy generation and power-grid security and the like, but they are the exception. The best way to advance your career in the Labs is to move around between departments every several years, so you will be limited in your options to do that if you have reservations about defense work.
Pay is good. I make $145,000. That is low for Sandians with 12 years experience I think, but have had some atypically low points in my Sandia career. You can make more in 'private industry' (other defense contractors) even in Albuquerque but you will lose work-life-balance
Benefits are very good. 3 weeks paid vacation. 2 weeks unpaid. Flexible work schedule: normal hours, or 9/80, or 4/10. Generous 401k match, plan supports roth mega backdoor, HDHP+HSA available. Good WFH was slow to arrive, but corona fixed that. Nobody has ever disturbed me on vacation or implied I should not take one.
Location is ok. Abq is high crime and NM is a poor state with poor outcomes but it is very rugged beautiful. Don't knock LCOL, it provides wonderful peace of mind, and the bad parts can be easily avoided, but maybe you won't like it. Relative to NM, Los Alamos is outlier with very good outcomes (crime nil, public schools among best in nation), because it is a place that only exists due to LANL. Sandia CA is option but while Sandia Abq pays well for Abq, Sandia CA pays very bad for CA. Californians often poached by FAANG.
Clearance means govt will look up your ass, and often. You must report all 'meaningful' foreign interactions, including friends/family. Investigations occur every five years. Random drug tests: get a phone call that tells you to go to the medical facility and piss today, or you're fired. 'Forgetting' to pick up your phone only holds them off for so long. Some (rare) clearances have worse requirements: must report all dual citizens not just foreigners; random polygraphs; must ask permission to leave the country; but again, these are rare. Easy to opt out of such a clearance and it will hardly limit your opportunities at all.
Project work varies. Nuclear is the mission but Sandia has expanded to wider govt contracts. Nuclear weapons, non-nuclear weapons/military, CIA/FBI/NSA partnerships, all are possible. If this is against your morals, there are other options, but probably better to work elsewhere. Would be like working for Google while hating ads. Myself, I am okay with thise things. I have not encountered govt abuses, nor any projects I consider inherently immoral, but perhaps I am naive.
I have had good projects and bad projects. Sandia is broadly 10+ years behind the curve at software engineering. There are pockets that are better, but eg version control is spotty in some places, lots of crufty old codebases. I think a symptom of being primarily an elec/mech/chem eng shop since the 50s. But there are 12000 people and if you find the right department, combined with the job benefits it is heaven. I have, and will stay till I retire. It's the perfect job for me.
You might also want to check out the US Digital Service, which might be more aligned with traditional SWE skills.
My daughter really wants me to work for NASA.
My point is just that this sort of work has a “wow” factor to it.
It's likely that you'll either be required to or pressured into obtaining a security clearance. Have a look at Standard Form 86 (SF86) and see if you're fine with these kinds of questions. Depending on how comfortable you are with this line of questioning, do note there will be more digging especially if you've lived abroad, had prior drug use, had prior legal issues, or a variety of other items that pique their interest. Failure to obtain a clearance is grounds for termination in most circumstances.
A clearance comes with certain responsibilities. You will have to report certain kinds of travel even if on vacation. You will have to report certain kinds of contacts even if not work related. You may not be allowed to freely publish even non-technical documents or books without prior approval depending on the level of clearance. There is a way to accommodate the lab in a way where you can mostly live your life freely. Most of this is just paperwork. However, you do give up some of your autonomy.
There is a proprietary innovation form they will push you to sign that will assign to them all innovations even on your personal time without company resources. Likely, this is not valid in California. Not sure if this is currently negotiable. They were pretty insistent in the past.
Pay for a starting Ph.D. was a little over $100k about 10 years ago. It capped out for most people at around the $130-140k region after 10 years of experience. Likely a bit different now, but not tremendously so.
While it depends on the group, internal politics heavily favors pedigree and degree level. Meaning, you are treated better if you have a Ph.D. even if it's not really necessary for your position. I'm not sure I would consider a job there with less than a masters. Practically, this means you're more likely to be the PI with a higher degree. More likely to be selected for promotion. More likely to be able to move to management if that's of interest. More likely to obtain internal research funding. To be sure, I don't agree with this, but it's a reality of the culture.
Things like time card fraud are taken very seriously. Meaning, if you work extra hours, you will be compensated for it either in terms of flex time or pay with overtime. Management doesn't like to pay overtime, so this means that you'll likely be out on time most of the time.
It's a professional workplace where people come to work and then leave. As such, no alcohol on the facility, no gaming tables, or sleep cubicles or cutesy architecture. It's not that people are completely serious the entire time, but they are professionals and there is a pretty strict separation between what's considered business activity and personal activity.
The research projects are great. There's easier access to grant money than in academics and it provides the opportunity to work in areas that are not necessarily commercially viable, but have broader impact. Getting access to the money either means being good at proposal writing, and having the right pedigree, or making friends with someone who is and likes to farm out work to you. When I say money here, you're not going to make anything more on the grant. It means being able to buy out your time to work on this particular project.
That's already probably too much, so I'll stop here. In short, I think they're a great place to work if you can fit into the culture with them. They tend to play by the book professionally and tend to hold by their agreements with you. However, there is a lot of paperwork and you will necessarily give up some of your personal autonomy.
I’ve worked with people at FNAL, LBNL/NERSC, SLAC, JLab, LLNL, BNL, Argonne - as well as NCSA, CERN, NASA, INAF/IN2P3, tons of universities and NSF research facilities.
I get paid now, but it’s because there was always chances to learn and challenges out in the national lab systems.
my tech complaints are that the existing IT people in the national lab system really are old school. Most computing is done in batch farms/grid, and if your not that you have to be creative for your resources. Cloud isn’t really something that gets used. Build systems are ad-hoc. Lab overhead rarely pays for anything software for the purposes of software development. People are generally smart but really conservative to new tech. The NNSA programs are a bit more flexible than department of science programs (Sandia/Los Alamos might be most famous). On the office of science side there’s a lot more cross collaboration between labs.
Simple things, like a $20/mo cloud service, are almost impossible to pay for. Expensive things, like a $400k purchase from Dell, is a piece of cake. Nobody is probably going to provide you a k8s cluster or anything, so running your own stuff/third party things is painful. Every national lab has a stupid opinion on which container tech to use and it’s never just docker or podman. Hope you like Singularity and Charlie Containers.
That said, I still miss it. I will probably go back at some point. I don’t miss having to run so many things, fight central IT on small things, deal with purchase. I miss the people. Age distribution is bimodal, probably peaking at 27 and 55 would be my guess for many labs.
Remote work is probably okay with Office of Science jobs, depending how close you might be to hardware. Probably not the case with NNSA work.
It’s easy to write/collaborate with software as open source among all the California labs. LLNL is the most impressive there by a large margin. Berkeley software is under BSD usually.
I was making 165 ish and now I make about 350k total compensation. Senior software would probably be around 145-185 at the very highest in bay area. If you can be an architect on a significant data intensive experiment with 200MM+ funding you might be able to get 200. Might get a 20k boost at LLNL if you are under NNSA related things.
1. Extremely high bureaucracy, very poor facilities management. Expect a terrible shared office in a temporary building with asbestos. Dont expect to be reimbursed for your travel expenses for like 3 months, and you're very likely going to have to travel for conferences (i.e. fly to DC to make your sponsors happy).
2. The old joke around LLNL was, "hey, you know how many people work here? About half." Half the people you work with will be the smartest most productive (and friendly!) people you've ever met, the other half are also smart but have realized that their productivity has no bearing on their advancement and so have decided to not give a fuck. Since hiring is such a nightmare, its very hard to get fired. From a program managers perspective, they'd rather keep people around who massively underperform then fire them since getting a replacement could take years. The upside is that if you want to get paid to do nothing, this is an amazing place to work!
3. You dont need a clearance to get hired, although it helps, you get to do the incredibly invasive FBI-agents-knocking-on-your-moms-door clearance process after starting, and then again every 5 years for the rest of your time there. The upside is that you get to make really fun Qanon jokes with all your coworkers. Be ready for random drug tests as this is a Federal facility and Cali's pot laws have no affect. Eat a gummy at a party? kiss your career goodbye.
4. If you're cool working on nuclear weapons, you're set for life. If you dont want to operate the gas chamber at Auschwitz (how I see nuke people), then your funding will perpetually be in danger, and you will likely spend more time chasing grants than writing code. (Nearly) Everyone you work with will be totally OK working on a tool with the express purpose of killing 100MM people. I had friends/co-workers who couldnt talk to their spouces about what they did during the day. The nuke people are also very enthusiastically pro-america in a creepy way that always set my teeth on edge. All diversity/inclusivity programs here forcibly killed by the GOP, which leads me to:
5. Politics affects everything. Government disfunction is annoying enough, but if there's ever a fight over the federal budget or a government lockdown expect it shutdown your work as well. If you work on something politically sensitive (hello climate program!) expect your funding to be on the chopping block. Real shit, when Trump came into office my entire program changed its name to exchange "climate" with "earth system science" to try to run under the radar.
6. (LLNL Specific) Livermore is very expensive (about the same as the rest of the Bay, but still pretty expensive by anyones measure) with none of the things that make the Bay nice. No BART stop and massive traffic means more than 2 hour round trip to Berkeley and back. Your Bay friends are not going to come out to visit, and in terms of travel time Sacramento is closer than SF. Unlike the rest of the Bay, its temperature is not regulated by the ocean so it regularly hits 115 degrees for weeks on end during the summer.
All that said, I still (mostly) enjoyed working there. The reason I left is because during covid I moved to SoCal and after the climate program got its funding reduced the HPC/Nuke people who wanted to hire me onto their team wanted me to come back into the office (there are certain terminals for accessing classified material that are fixed in place and cant move).
The pace is definitely much slower, and in fact you'll find a lot of people just barely doing their part and kind of hiding in the woodwork, so to speak, which annoys some people to see. You also see that in big corps too of course.
I was given an enormous amount of responsibility for being right out of college, which was basically just determined by my boss who had won a grant to do some specific type of research, so he pretty much decided how to spend the money. In retrospect with more experience, I was underqualified for the job, but I had a lot of fun and learned a ton.
My boss and his colleagues had to account for their time in something like 15 minute increments, because their time was billed out to their projects. Now, I don't think they actually tracked their time that closely, they more realistically were probably just billing time according to the money they had in each pot to keep the projects going on time.
Which brings me to my next point - the labs as far as I could see run as a form of contracting business. They compete for funding from other agencies, like the military, DOE itself, BLM, etc., to do either basic research or applied research. So grant-writing and competition for the next grant is very important. Each principal has to have many grants in the pipeline at all times to make sure their funding doesn't dry up at an inopportune time. This can be stressful I think for people, depending on your role in this process and your personality.
In some cases you'd just be working for someone else who had secured all the funding for many years, and in that case you'd never have to worry about that aspect.
I see some other comments that not having a PhD means no one listened to you... that wasn't my experience. My boss only had a masters, I had a bachelors, and we pretty much did what we pleased, and decided how to do it ourselves. I'm sure it totally depends on the area you are in and your working group though. I think our group was small enough that none of the PhD scientists gave a damn what we were doing, plus my boss had gotten the funding with his colleagues (all of whom were masters degrees, no PhDs in the group at all), so no one could really say boo to them.
I ended up leaving because if you don't at least get a masters you get capped really quickly in what you can do. In my case they didn't have a permanent role for me at all with a bachelors, I was on some kind of extended internship of sorts. I decided to go into industry instead of going for a higher degree.
* access to large interesting tools and projects
* Big big budgets that often take good long looks at their decision long term making
* Generally very smart people around and interesting community events unlike working at a corporation.
* Perks and benefits well above and beyond working at corporations (often in lieu of higher pay that corps would offer)
I worked on the LHC and RHIC projects on the data processing side and it was pretty great. The laboratory itself was very run-down and not maintained well in many areas. Conversely because you're often shortstaffed you get enormous opportunities to play with various aspects of technology and head up interesting projects.
Day to day it was a lot of coffee breaks and discussions about where to go for lunch that day. There's a 'hurried' aspect that probably exists at one of the FAANG's.
Last bit of advice, most labs are either DOD or DOE managed facilities that are often managed by a large consortium, in BNL's case it was Battelle. They are often run this way to keep workers from being Federal workers with more rights and perks. Different labs are run by different consortium's, something to be aware of.
