Why are JavaScript dependencies so messy?

I think there are a few things here that are kind of unique to the js/web platform.

- JS doesn’t have a proper standard library and so you tend to have many more dependancies in a project than with other languages.

- Because of that you then have a very large tree of dependancies, often with incompatible version requirements of the same package.

- Due to the nature of needing to “bundle” your code for distribution on a “slow” network, a lot of optimisations (tree shaking, code splitting, async module loading) are required to minimise your bundle size.

- You say “We only have a few module types”, that’s two more that every other language. And they are incomparable in subtle ways.

Those and some other issues are completely unique to the web platform, they are some difficult problems to solve.

However, I think the current generation of tooling has finally got there. Vite, with its esbuild and Rollup backend is bloody brilliant. If you use a framework with official Vite support it just works.

I'm sure I'll get downvoted for this one but quite frankly it's where the PHP developers all slithered off to. It was inevitable.

I avoid to the point of refuse to work with any JavaScript platform at all. They are a universal shit show.

Edit: honestly I stand by this comment. I literally have spent hours this year dealing with fucked up messes in nodejs, package security issues after repos were hacked and code made it into NPM that displayed banners on commercial sites. My comment is toxic yes but quite frankly both communities are as well (PHP and NodeJS) and utterly earned that ire and discontent. The fact people lean on the stack and produce stuff that handles critical aspects of people's lives is utterly frightening. So yeah toxic, I don't care. Feel free to be annoyed. It's still a turd, just been rolled in glitter.

JS is retracing some of the same steps of Python. They are both a fun language with not a lot of constraints and a lot of capability.

What this does is create a large community of enthusiastic beginners led by few (prone to follow trends by authoritative members) so you get some experts but a lot of beginners and beginners love libraries, especially ones written by the expert contingent. The JS stdlib is not as strong as it could be, nor is python, so libraries in some cases make a lot of sense but for beginners it is easy to rely on libraries for everything.

Eventually some of these beginners turn into skilled programmers and either stick with just JS or Python, and then the other subset moves on and tries other languages that address some of the perceived 'shortcomings' of JS/Python. What you find is that a lot of the ecosystem becomes that beginner crew and a repeated brain drain of the other 70% who move on.

Dependencies is different from bundling. Javascript's dependency management is an absolute dream, and you can have a newbie up and running with a consistent environment very quickly. Compare that to Python, which still doesn't have a declarative package manifest format (leading to extremely slow/inconsistent package resolution), or a mature "lockfile" for deterministic installs. npm has both of these out of the box. You can even assert what versions of npm or node you need in your package.json to make sure you're using known-working versions.

EDIT: Turns out Python just finalized a pyproject.toml format in 2021. Of course, this doesn't really help much until every package out there migrates. npm has used the package.json format pretty much since day one. And there is still no standard lockfile, other than dumping the output of `pip freeze`.

Because it's a language and ecosystem built by mostly designers and unicorns not backend programmers who rely more on logic and functionality rather than colors and rainbows. My 2c.

I tried to compile a rust project the other day and I realised I had to download hundreds of dependencies. I guess strong typing helps but dependency hell is just common for any moderately popular platform.

I hated NPM... then I started trying to use some Python projects.

Now I like NPM.

Dependencies are hard because dependencies are hard.

There’s no easy dependency management system.

Bash - the most interoperable language in existence - contains its own version of malloc because some systems used to need that (back 30 years ago).

JS has an extremely uncertain runtime. Is it in a browser? Engine? PDF?

Because JavaScript itself is nothing but messy and a lot of new programmers start with it and somehow learn it and follow in the same vein

1. backward compat. most still use commonjs over ecmascript. some even use amd bundles.

2. design by committee. most large projects have users that have edge cases. devs sometimes create messy apis just to cater to a larger group of audience. notnecessarily bad, sometimes the product just grows, sometimes its for security reasons.

3. incompetence. sometimes you dont need a library. sometimes their implementations suck in terms of readability, security, space and time complexity, maintenance.

4. code golfing and not using linters. these 2 are pain in the ass.

Just don't say that you aren't fan of creating new app with 17 vulnerabilities out of the box.

Things change. Web development has a lot of incidental complexity that these tools help manage. Eventually tools are overcome by the complexity.

Parcel in particular is a clever wrapper on top of several other tools, so I’m sure when their dependencies break they pay the price.

I think some of them are also victims of their own success. Lots of these tools have a happy path, but then they gain more and more users who want various exceptions to the happy path.

It’s messy because it actually works. If JavaScript couldn’t scale indefinitely then it would never get to the point where it is messy.

npm introduced the concept of independent transitive dependencies. This is very unique when first introduced. Most other dependency managers require the importer to reconcile “conflicting dependencies”.

Dependencies are a messy, human, complicated problem.

What would an ideal dependency manager look like? It could read the source code of a library and the overall program, determine what sections of code / apis / interfaces are actually used, and perform that recursively for each interfaced library with the libraries it interfaces with.

Or the compiler / programming languages would require lots of annotations to aid this, possibly both human and compiler. Imports and interfaces would need to be very fine-grained.

Now add in operating system compatibility, versioning ...

And then what if library invocations aren't a static/explicit reference? What if they are "plugins" whose module name is in a configuration file and is dynamically loaded?

Oh, no small thing, what if your language doesn't even have any degree of strong typing, so analysis of apis cand api invocations can't easily determine the types of the objects, methods, signatures, parameters, data being used to be invoked...

Javascript has been in the worst situation for this: competing solutions, rapidly evolving language, rapid churn in frameworks and libraries and interfaces.

Javascript also didn't have a strong server-side / cli toolchain origin and centralized authority. It was a browser language repurposed by dozens of vendors for other purposes, almost all without regard for an overall ecosystem. It was subject to major corporation monopoly battles.

Javascript also has a "neckbeard" shortfall. It's a language ecosystem where those people with decades of experiences aren't generally present.

None of that is a good formula for effective package management at an ecosystem level.

Python is kinda similar, at least pip provides some centralization, but good lord does python suck for package management and dependency resolution. Python started as a traditional UNIX-y language for scripting alternatives, so that provides a more solid foundation.

But look at linux package management. Slow, huge, error prone, and ultimately requires containers to solve with reliability.

Yes it's terrible but then again alot of package managers and dependencies are, however i think the sheer volume in javascript, and the accessibility for anyone to post to npm has it's own set of problems, one thing i noticed is that interfaces constantly change in NPM packages compared to others i work with, its very hard to not ending up locking to a specific version because of API updates. If this is due to JS community or the ease of access to create packages, i dont know but it has something to do with it.

That's because JS with its dynamic typing is great for developing code quickly, but not so great for maintaining and evolving it over a longer time frame.

Then there's the lack of the standard library, which means your code has to depend on code of random people who may or may not know what they're doing.

Also, because it's the web where patching code is quick and easy, people don't seem too bothered with quality.

I've chosen mobile development myself and fortunately things are a lot more sane here.

Probably the only real reason I avoid JavaScript is this mess.

Maybe if you maintain an dependency you can feel important.

The different module syntax is the problem. If we just had the standard import/export syntax from the beginning I think 99% of problems would be avoided.

Going to try and give a reasonably systematic answer to this:

Firstly: the title of the post is different to the content. The title asks why dependencies are a mess, the content cites problems with bundlers. These are entirely distinct topics. Bundlers don't handle dependencies and package managers (mostly) don't concern themselves with bundling. They're different problems and have surprisingly little in common.

So the above breaks it down to two questions:

1. Content: Is javascript bundling a mess?

2. Title: Are javascript dependencies a mess?

To answer question (1) let's first look at it comparatively. Are other language bundlers a mess? Obviously they don't have any as bundling is a client-side problem and javascript is the only client-side target. So it's hard to gauge this. Maybe bundling is a fundamentally hard problem that will always necessarily be a mess.

Subjective opinion: I think there are obvious ways bundling could be done better and it surprises me they haven't been done but speaking from a place of inaction I'm reluctant to be too critical here.

To answer question (2) let's look at it comparatively again: are javascript's package management and library ecosystem more or less of a mess than Java's, Python's, PHP's, Golang's, Rust's, etc.? Some may disagree here but I think the answer is objectively "no":

- The Ant/Maven/Gradle/Bazek/sbt ecosystem is absurd complexity, and I've seen plenty Java .m2 dirctories worse than node_modules. I think Java must only escape the criticism JS gets here because every Java dev I've met is so reliant on their IDE they don't even know .m2 exists (same goes for ObjectiveC/Swift)

- Python is both better and worse than Java. On one hand, the manager ecosystem is even more unapproachable: the setuptools easy_install pip pipx conda poetry venv pyenv nightmare isn't even helped by consistently demarcated config files in your repo root: good luck figuring out what to run depending on whether there's setup.cfg setup.py requirements.txt pyproject.toml etc. (if they even exist at all - pip not requiring project root dep config is why so many Python readmes just have long handwritten accounts of which deps you need to install manually). On the other hand, python's actual dependency tree handling errs on the oversimplified side. Have two transitive dependencies on separate incompatible versions of a single library? Tough shit, rewrite your code.

Go seems to be leaning too heavily on Git and falling into the trap of modelling their package management system on the implementation rather than modelling the implementation on user needs. It's not terrible though, but that's perhaps due to its age.

PHP is probably the least messy, and Rust - while making some mistakes - is new enough to be following some good modern examples. But overall NPM is by far one of the LEAST messy package management ecosystems.

I feel your pain on this. As an occasional NPM library author, I feel that Typescript can help with this, although NPM itself is not structured in a way that compliments Typescript: it’s awkward to write in Typescript, publish to javascript, then consumers fetch your library and go back to writing in Typescript.

Letting the versions of your project dependencies float without control guarantees some headaches.

And, of course, the Fast & Furious dependencies development lifecycle has some side effects

> every once in a while a new bundler comes along

I think the answer is (at least partly) in the question.

The necessity of rewriting the world in javascript as richer browser-based apps took off and native browser plugins went out of style meant everyone and their cousin needed (or "needed" https://xkcd.com/927/) to implement some library or functionality from scratch themselves. On top of that, the npm default of publishing everything publicly meant that what once would have been an internal utility or wrapper became a de facto public api.

That behavior ingrained itself into the culture and community around the ecosystem far more than is good for it, imho.

Messy? I invite you to try some C