HACKER Q&A
📣 PaulHoule

Meta-build tools for complex projects?


Most of the systems that I've worked on in the last 10 years use a web-based front end. The "Project" on a project management level involves delivery of both a Javascript application and a server written in (say) Java or Python that the Javascript application communicates with. So there are at least 2 "projects" in the sense of an artifact managed by a tool like maven, uv, npm, etc.

Practically these things usually have multiple Java projects.

I have a system with about 15 "projects" in the small sense that supports roughly 5 "Projects" in the big sense. There are two back end servers and several front ends that work with them. If I am working on one "Project" there are maybe 8 "projects" that are potentially involved with the build. Some of the Java projects have liquibase in them and get extra build steps.

I have a Python script that not only runs the builds in sequence but can also automatically merge changes incoming from the develop branch. It's pretty ragged, for instance it does not topologically sort the dependencies but depends on a total order I wrote by hand. I feel like it improves productivity because I can push one button for a build and not have to wait for a number of steps to happen.

(Earlier I had a JavaFX tool that did the same for pure Java projects)

I'm wondering what the state of the art in this area.

  👤 zahlman Accepted Answer ✓
I don't know if there's really anything like "state of the art in this area" - certainly I haven't heard of it - because it isn't very well defined. People who do packaging tools are generally expecting to stay within one language's ecosystem, and it's not even clear what concrete requirements or expectations people might have from their tools, in cases like yours.

That said, your current approach sounds exactly like what I would try. Orchestrating build processes in Python is right in my personal wheelhouse - and Python's reputation as a "glue" language is well merited. (There's another recent submission https://news.ycombinator.com/item?id=42303384 implying that I'm not alone in this.)

In principle, nothing prevents you from preparing wheels that contain only Java or JS code, hosting a private package index that implements PyPI's protocols (see e.g. https://stackoverflow.com/questions/18052217 to start) and then leveraging Python ecosystem tools like Pip or Conda. But maybe that's more complexity than you need. And of course you'd have to have a plan for using that code once it's installed in a Python venv (or Conda environment). (If you do take this route, and share my distaste for Setuptools, you might one day find some use in the build backend I'm currently working on. It's definitely not ready to Show HN yet, but I do have this sort of use case in mind.)

For what it's worth, there are many libraries on PyPI already for implementing topological sorting.

Another thing that occurs to me is that if you want the traditional C Makefile experience (describe dependencies; deduce what is out of date and run specific commands to rebuild it), there are more modern options for that. Start by checking out Meson (https://mesonbuild.com/) and Ninja (https://ninja-build.org/). They aren't what I would have thought of as "meta-build tools", but they do get used for multi-language projects (like the SciPy stack) and might be what you're really looking for.