Anyone using Bazel at startups? or migrated to/away from Bazel?

At a startup, you're going to rewrite you're entire product multiple times because you don't know what it actually is when you start. Your job is to get shit done as fast as possible to prove out the idea before you run out of money, and to make a demo that can convince people to give you money to actually build it right with all the lessons learned.

Bazel gets in the way of this use case. It will help your small team not break each other's builds. But you'll burn a salary on someone as the "Bazel person," they're going to write a bash script or make file for everyone else to use anyway, and when you inevitably rewrite everything from scratch the third time all their work is going to be for nothing and they'll spend days getting it back.

Just write a bash script or python script that does what you need for today, and plan to deal with it later. The only thing startups can afford is tech debt.

Another way of thinking about this is that your product is the software and not the software to build the software. Don't waste time thinking about the tools.

I worked for a company of ex-Googlers that used Bazel with Python/Django a few years ago.

- It copies your files into a temporary directory to run them and that directory is named an obnoxiously long hash digest, so it effectively takes a giant shit in all your stack traces. When you're looking at an error message, half of your field of vision is taken up by line noise of the same random hash over and over again and since it runs your Python the first few frames are just helpfully telling you "this code was run through bazel". I even wrote a script that scanned my clipboard for Python stack traces, parsed them and removed this noise to make it easier on my brain to search for the actual line with the error.

- Every import you write you have to write in two files or three if you have to export it. It's double the typing work and finding the target is a pain because there's no jump to definition of the Bazel target so it's even more typing to find it.

- I wanted to use pytest and it was much more work, I don't think I ever got it working. Normally there's no setup you just run "pytest" and it discovers all the tests and runs them.

- Your code doesn't start running as quickly.

- You have to type way more characters to start your code. I had aliases for every endpoint.

- There was something with tests where if the error log was too long it would just say "there was an error, please find it in this randomly-named file" and when you're debugging a test whose error log is too long it's going to be too long every time you re-run it too. So running and seeing the error is now a two step operation, double the work.

Maybe some things have changed or I misremembered something, but overall it was the part of our tech stack I really hated. I just wrote software, I didn't do any ops or any sort of deployment or touched live systems at all really, so I honestly have no clue what it actually did for us it was just pure downside for me. I would never use it.

I worked at a startup using bazel, and had a strongly negative experience, for the following reasons:

1. Bazel requires non obvious, continued maintenance. Bazel is a theoretically hermetic build system with universal caching and near constant build times. The sticking point is that it only has those properties if someone sets it up correctly and maintains its infrastructure. At our company, no one set this up, and as a result build times ballooned and local development became a mess of bash and docker. And even if someone had set this up, who would maintain it? We would have had to expand the infra team, and even a couple of people extra can be a lot for a startup.

2. Bazel also has support for every language, which is theoretically true. In practice, this support is often incomplete, out of date, poorly documented, badly supported, or some unfortunate combination, because the (language)+bazel communities that support the docs, tooling, etc are often orders of magnitude smaller than teams that work with (language). So teams that use bazel are forced to choose between two bad options: integrate things the bazel way and rely on the relatively small (language)+bazel open source communities and tools of questionable quality, or use the much much larger (language) open source communities and hack the tooling in. We did the former, and it was hell, because we a) didn't have feature complete community tooling and b) upgrading the bazel version required every team at our company that was using bazel to stop working and simultaneously check trial bazel versions for breakages.

3. Bazel makes hiring and onboarding harder. In order to fix the above problems, you need people that know bazel. Hiring someone that knows, say, python is hard enough. But python and bazel? The hiring pipeline immediately gets reduced, which means you either have to spend more time searching for qualified candidates or more time onboarding candidates who don't quite fit all your criteria.

All in all, I would recommend against bazel unless you: 1. Have or can hire a preexisting community of knowledgeable bazel developers who will make up a sizeable portion of the company. 2. Have full executive support for multiple years while the infrastructure is being created, and executive support for maintenance teams thereafter 3. Actual, company-wide problems with builds that justify the former two criteria.

Also ex-Google engineer and founded a bunch of startups in my time away.

I tried Bazel when it was first released as open-source but migrated away from it pretty quickly. It was just too complex for what it brought to the table. In a startup you should only be doing things that directly validate your business hypothesis, and futzing around with your build system is not really one of them.

I found it much more productive to use the native package managers and build systems that go with your language, eg. npm for Node, pip for Python, Gradle for Java or Kotlin.

Don’t be the Google hire that makes everything complicated. Be prepared to be told no, or to bring a better argument.

We use Bazel at work and it’s sufficiently complex to write new rules. I’ve seen NX touted as a more approachable equivalent for smaller teams, maybe check it out?

Bazel is as much an HR Department (licenses, supply chain source of truth, entry point for sec teams to add stuff to, auto generate jiras for out of date deps etc) as it is a build system. If you mean an actual startup (and not startup as in airbnb), then use the native build systems until a standardized system like bazel is needed.

Also an ex-Googler. At a tiny startup of 4 engineers atm.

Briefly looked into Bazel, but settled on pantsbuild since Google seems to have just thrown all the language support over the wall to the community.

Being able to trivially setup CI on Github has been very helpful and important to me.

Being able to easily add targets and package things that require multiple microservices has been nice.

People say you can use the native tooling, but what even is the native tooling for python? It doesn't have a standardized packaging tool unless you count docker. pants' support for pex was important for me until we eventually adopted containers.

Folks can (and did) get the native packaging wrong, particularly when they were not super familiar with the language they needed to pick up on the fly.

Having to deal with an abstraction layer over the tooling has introduced some of its own headaches (eg for docker), but mostly for advanced features.

In the end, I'm pretty happy, I see it as an investment in CI, and the CI has caught a bunch of real bugs people pushed.

I think starting early here has been good, since the lift to get onto it has been small.

I worked on a project where I used a Google framework that used bazel and it was a terrible experience, even though in the end I made it work. My first approach was to integrate the framework somehow into the build process I was used to (Makefiles, cmake etc), but then quickly found out this was going to be nearly impossible due to the way bazel works, so in the end I had to use bazel to build my project and to be able to add the framework as a dependency. I still had to hack into the framework bazel tree here and there. I lost a lot of time learning the quirks of bazel and in the end it contributed absolutely nothing and instead of bazel, cmake could have been used, a solution most people are familiar with. I think Googlers are generally on some kind of quest where they need to prove how smart they are by inventing all kinds of quirky weird new kind of "wheels" instead of using existing wheels to build a car, which causes problems for people who do not work at Google but are somehow forced to use Google's solutions. I am wondering if Googlers are aware of the fact that by using/inventing these kind of solutions their skill set becomes so specific they can't leave Google anymore.

I have tried twice in the past 5 years or so to convert different projects (mostly Java) to Bazel but failed miserably every single time.

I do not have notes handy so I don’t remember exactly what happened but my impressions after a week or two invested trying to get it running were a) that our projects had too many unsupported things (dependencies, flows) that would require a huge effort to adapt; and b) that it would take way too much effort for our small team to convert the 500+ repos we were building to release our software.

I have always dreamed of a time in which I’m able to fin a tool that fills this gap. Man, not having to build the same code that hasn’t changed over and over locally and in pipelines sounds so appealing.

This article might provide some insight:

https://earthly.dev/blog/bazel-build/

My thoughts, as someone who has done a lot of build engineering, but that has only a small amount of experience with bazel:

If you are working with languages that are well supported by existing, maintained bazel rules, maybe. Otherwise, the answer is likely "no". You do not want to take on the burden of maintaining significant rules, or figuring out how to mate your ecosystem of choice to an opinionated build system that nobody else in your ecosystem uses.

If you can manage with something easier to setup, it probably makes sense to do that instead. My general experience in startups is that you have to "pick your battles" constantly. You don't have the luxury of infinite time and resources. Work on the build system has a high opportunity cost.

Most developers and other people who aren't build nerds care about the answer to two questions in regard to builds:

1. Is the build fast enough?

2. Is the build working?

That's it. This is frustrating when they don't understand the other important aspects of build system quality, but it is also, admittedly, somewhat of a pragmatic perspective.

There is much leeway inherent in the speed question. If the build is slow, people can wait. Do they have any other choice? There is plenty of other work to do while they wait.

If the build doesn't work, you, THE build guy, are going to feel the pressure to get it working again quickly. This often turns to bodging, and eventually to something resembling life support. However, nobody will care how much of a brittle, knotted mess the build code is if it still manages to mostly work.

To put it bluntly: A startup company can probably survive with a shit build system. And that might even be the right answer depending on the context.

I would personally be perfectly happy to get paid to set up a beautiful, apex example of a build system for a startup, whether it makes business sense or not. My professional integrity just won't allow me to recommend that path unless I think it's justified.

I might recommend to start with a script, or a makefile, or a ninja generator, and see how that works out first. After all, that is how Google did it for a long time before they decided to create Blaze.

I've used pants at a small company <10 engineers and worked at a company migrating to Bazel with 200+ engineers. Bazel used to take a dedicated engineer 3-6+ months to bootstrap the full setup (creating the build files, seeing up the Bazel server, updating CI to use Bazel correctly including caching) especially since the docs weren't great from what I remember. I suspect that the time could be cut down by now especially if someone experienced with it does the implementation. Given the amount of time necessary for the setup, it typically makes sense for companies with monorepos with 50+ engineers although existing expertise would change the equation.

A few years ago, I was involved in a team with less than 10 engineers, we decided to switch to bazel in order to improve the build times, specially the CI runtime.

On the good side:

- bazel forces you to have an acyclic architecture, still, this can feel annoying because every module needs to specify its dependencies.

- bazel does concurrent builds pretty well, optimizing the build times.

On the bad side:

- It took a considerable time to set everything up, we thought to be done but no, there was a weird stuff coming every week and most of the team ended up waiting for the bazel guy to fix it.

- IDE support was poor, in theory, we could just use intellij but most of the times it required help from the bazel-guy, in my case, I got frustrated about the needed maintenance and just accepted that an IDE wouldn't work. - Given the lack of IDE, it means that we couldn't easily execute a specific test from the UI, having to keep notes for the cli.

- In theory, you could use a shared cloud cache so that the builds are very effective but we could never set this up. - talking to engineers from other companies, they admitted that the bazel setup was hard and took a considerable amount of resources, they ended up writing many internal tools to get a effective usage.

Given that you plan to create a startup, do you plan to hire engineers? then, be prepared to the bazel-guy.

Also, as a startup, I bet your code will be small enough that the build times won't be very different with normal build tools.

I'd personally not use bazel unless my company offers a dedicated engineer just to maintain this integration, I think its overkill for small companies.

Our CTO is from Google and likes Bazel. It's awful to work with and is slowing down development tremendously. The worst part is packaging up third-party dependencies.

It just really depends on what the startup is doing.

If it’s a bunch of Python with a C++ extension (or mostly JVM or whatever), heavy build stuff is probably overkill.

If you’re in a polyglot environment with mountains of vendored native code targeting multiple cross-compilation targets and need the flags tuned just so?

Then Pants, Bazel, and buck2 all have pros and cons.

If you actually want to go down this rabbit hole, then use Nix instead. It is an open community project, not a corporate monstorsity.

I would stay away honestly! I am ex-Meta and even when I was working at Meta my entire team struggled mightily with Buck despite having access to all our internal resources!

When it works you almost forget about all the pain... But then once I took a peek at some of the custom build rules that I was blindly copy pasting and was blown away how complicated it all was. Thousands of lines of build code...

i have a lot of recent experience with yocto/bitbake but none with bazel. do they have overlap? could they be used for some of the same things?

I've rarely had worse experience with infra tool in a big company.

Can't imagine this layer of overengineered complexity can boost a startup, but can easily picture how it slow it down.

Your thinking is obviously distorted by some sort of google mentality.

Put your foot back to the real world please.

The world where you have competition, world where people need to get job done quickly.

Not in most technically sofisticated or mathematically cool way to brag about, but quickly and with as little overhead as possible.

What about buck2?