Why Poetry did not become a mainstream package manager for Python?

Because venv/pip/requirements is good enough for most cases. Most developers are introduced early to these tools and learn them good enough to work fine for them.

If there's no real problem with the existing tools, then people will not change their tested and trusted way of working for some fancy new tool, no matter what that new tool promises!

There's an old engineering proverb about that: if it works don't mess with it.

It is complicated I think.

For one, I think competition, there is Kenneth Reitz' pipenv and then of course conda's environment.yml. I think pipenv lost the popularity contest but somehow lead to a situation where people wondered on which horse to bet.

Conda also takes a lot of market share esp. in the machine-learning deployments while also being not a 'perfect' tool.

Pretty sure we will see more poetry in the coming years.

Last but not least Poetry sometimes also feels like a weird tool. For example, when its dumping colourful stack traces on you in case of (invalid) inputs, etc.

If pip-compile would support dev and production requirements it would still be my way to go I think.

Poetry does need help. I created a thread here: https://github.com/python-poetry/poetry/issues/4160

my primary concern poetry's development stagnating due to natural causes while many projects rely on it. switching costs are high. moving from pipenv -> poetry took me days across all my projects.

they're looking for developers, but also help with managing issues on the tracker.

i also raised the issue of them being funded. I think one possible outcome is seeing if PSF's Fiscal Sponsoree program (https://www.python.org/psf/fiscal-sponsorees/) would be a fit. Waiting to see what their maintainers say.

Poetry is a mainstream package manager. Every company I know uses it.

And more importantly, the concepts it brings will eventually become Python standards (like pyproject.toml or the lock file): https://snarky.ca/what-the-heck-is-pyproject-toml/

The author of Poetry has an open Github sponsors account [0]. If you're using Poetry, or rely on it in the commercial context, I'd suggest investing into reliability and sustainability of your build tools :)

In my startup, I started to make sure that we allocate a small monthly budget to distribute between our open source dependencies, Poetry being one of them. It's always shocking to find how little support the projects are getting - the author of Poetry currently has 11 sponsors (with majority of those being individuals).

[0] https://github.com/sponsors/sdispater

I do think we're at a cusp where the poetry adoption rate may increase.

I avoided poetry for years because it doesn't have an option to skip the lockfile. We build images, so our dependencies are frozen anyway. (Now that we use dependabot, I'm willing to have a lockfile that's automated.)

I track Python packaging changes carefully, updating from procedural setup.py, declarative setup.cfg, versioning through setuptools-scm and now PEP517 and pyproject.toml.

But even now, Python ships with setuptools, pip and venv, and a system like poetry is not really a necessity.

But if you want convenient virtualenv management, editable installs _and_ PEP517, then you probably need poetry. I haven't found another system that can do all three.

FWIW, I placed my early bet on pipenv, but it had some significant bugs and development had stagnated (seems like it might be picking up again).

When we decided to move away, I looked at Poetry and pip-tools. Poetry seemed too different to me, don't remember specifics, but felt like too much would need to change. All I wanted was the ability to lock package versions.

Enter pip-tools. It was really simple, felt familiar, and just worked. So, that's what we chose and it's what we use to this day.

I think poetry is rally gaining traction. Poetry solves some problems in an easy way that are cumbersome to replicate with pip (like guarding against supply chain attacks with lock files) and teams are starting to realize that basic use of pip and requirements.txt no longer is enough to manage dependencies. I am starting to more an more projects adopting poetry and the uptake seems to look like the uptake I saw in FastAPI. Side note: I tried using pipenv, but is was just too slow for resolving dependencies. Curious what other people’s experience is on this with pipenv and poetry.

Nothing is holding it back. It is making progress but Python is a big community and poetry is still new. A new way of managing dependencies does not take off overnight (and nor should it! go slow, and break nothing!).

I can speak for myself— venv is good enough for most things and conda works for everything else (pandas/numpy/tensorflow on Mac m1).

For production code I’m more worried about minimizing surprises then elegance which means it’s really hard to replace something that works. Poetry only hit 1.0 in 12/2019, so maybe I’ll try it if the api remains stable for another few years. Contrast that to JS where we’re now on webpack 5 I think and there’s really no long term support available.

Because it’s unclear (to me at least) what problems these tools solve.

Pip apparently isn’t deterministic in its resolutions, but you can achieve determinism the same way Poetry does by creating a frozen requirements.txt if you absolutely must.

I’ve only seen dependency hell in badly designed projects, particularly ones where people use libraries to share code in an organisation (which frequently turns out poorly because maintaining libraries is harder than people think).

Poetry is also really slow. I’ve noticed on larger projects doing ‘poetry add’ takes nearly 10 minutes. This often happens when people use wildcard dependencies; for whatever reason the resolver looks through all possible version combinations, which can cause resolution to take exponentially longer than it should.

Lots of people have a process that works well-enough. There was confusion and bad experience with there being multiple options, e.g. pipenv (which came with its fair share of drama), making people wary.

Because `python3 -m venv venv && ./venv/bin/pip install * && ./venv/bin/pip freeze > requirements.txt` is good enough.

for a start, python’s package manager is pip. Poetry is the leading “pip but also more” tool, and has been for a while.

Can't speak for others, but in my own & work projects I'll use containers (not just for Python but other dependencies). That gives me the isolation I need (e.g. if I need to use a different Python version and/or set of libraries) and pip/requirements is adequate (maybe using pip-tools to keep the requirements.txt up to date). While you can of course set up Poetry in containers, it adds complexity and build time and doesn't give me much on top of what pip+containers provide.

That's an oddly worded question. Poetry is a "mainstream" packaging, dependency and virtualenv management tool. What makes you think it's not popular?

Tangent: Poetry can have very slow dependency resolution. It doesn't cut to the core of the problem: Pypi doesn't enforce clearly specifying dependencies. Many packages do, like in Wheels metadata, but Poetry ignores this, and installs each package.

There are some ways around it. For example, build an independent cache of metadata to use instead of Pypi directly.

For discussion on the topic, see this GH issue: https://github.com/python-poetry/poetry/issues/2094#issuecom...

This is now fixed, but for most of Poetry's life, it took some tricks to work on Python 3, at least on Windows; the workaround wasn't documented. So, it'd always assume Python 2. I can't remember the details, but tried and quit Poetry several times during its early life for this reason.

Poetry strikes me as about 90% of what is needed.

I use it all the time because otherwise I’d have to write my own build system.

I am not sure if poetry’s approach to solving dependencies is really correct (Personally I like downloading the metadata for the projects, SMT solving, choosing a set of wheels and installing them together. It only works for wheels since you can’t extract the dependencies for eggs w/o running setup.py)

Pip is almost right if you install all deps at once but it can’t update wrong versions of old packages incompatible with new ones you add.

Practically poetry handles cases that pip doesn’t.

My beef(s) with poetry are:

   A. It is bad hygiene to copy files by default into your package; they should quit pulling wings off angels and just make a /src dir.
   B. Poetry itself gets corrupted and fails to update.  Delete and reinstall always gets me back on the road but I was anxious about it at first.

I use Pipenv now for applications that I’m not publishing to PyPI. I like the workflow better, and find it to be much less buggy. Pipenv is not really involved in package building, though.

For libraries and OpenSource that I’m building actual packages for, Poetry seems nicer to me than setup.py. I’m trying it out now with one, partly to adopt the newer pyproject.toml thing.

Packaging has always been kind of an ever changing mess in Python vs newer ecosystems, and I don’t see an end to that.

I’d just do what works best for you.

Also, the general opinionated aggressiveness of the maintainer/project is a little off putting to me. Its a little weird in the community.

Normally projects don’t dog other projects on their page. Often they cite alternatives with maybe a statement on how the philosophies vary.

But that’s just my opinion and experience. I’m sure others differ.

I for one am happy with pip/requirements.txt and venvs, I guess if I wanted to publish packages I would look at a more comprehensive system, but for internal apps and scripts I don't see the benefit. I'd love to know why others prefer Poetry, though.

For me virtualenv+requirements is good enough. The only feature of Poetry that I miss is the ability to create wheels without a setup.py. But I don't create wheels when writing backend code. So I only use Poetry for working with packages.

Give it time. It's easily the best ome of the bunch, but python has tons of legacy.

I used it and disliked it. I can't exactly say why anymore, it was a while ago, but it just felt odd, and confusing.

I did not continue with it. Maybe the documentation was not written "to my tastes". I don't know how "mainstream" it is, but I suppose if it's initial taste has been better, I'd have stuck with it. I suspect there are others in my position, and hence, maybe it didn't take off as well as it could have.??

Is it just me: I used pipenv before knowing poetry. Pipenv mostly fits the bill. Then, everyone was talking about poetry so I gave it a try but the experience wasn’t that great. It’s either doing what pipenv could do or running into issues I found hard to understand. So now, I just use pipenv. And I think pipenv development just has been started again.

I’m a casual Python user btw.

I like to keep it simple. When I'm not working on a dockerized project I use virtualenvwrapper + pip and they do the job just fine.

Poetry has been usable for how long? Maybe two years? Python packaging has been a dumpster fire for that many decades. The developers who haven't left for languages with more sensible package management have been burned by the latest Python packaging fad many times. It is understandable that they would be willing to let others work out the bugs first.

I like to use it for applications, because of the lock files and easy virtualenvs / all in one experience.

However, I also maintain a number of packages with c extensions or other compiled components, and poetry has no documented way of building packages with binary components, it can only create non-any wheels.

People think package manager as a package downloader only and for that yes, PIP is more than enough.

The main problem for me is pipenv and pip doesn't solve dependencies version issues when you want to upgrade one or several packages. Poetry in the other hand never had an issue.

I looked at it a bit, but not long enough to really understand how it's better than modern setuptools w/ a fully declarative setup.cfg. I'm not sure I have whatever problems it's solving.

Its pretty good but you can get almost all the functionality with venv, make, and pip.

pip install -r requirements.dev, reference requirements.txt in it. It will install dev packages and the main ones.

A tangent...

Glyph† pointed out that, in Python, as an OOP language, we should objectify all the things, e.g. like what Pathlib does for file/path manipulation. (Pathlib was inspired or extracted from Twisted, IIRC.)

A Pythonic dependency/package manager library would be soooo cooool...

† https://en.wikipedia.org/wiki/Glyph_Lefkowitz https://glyph.twistedmatrix.com/

It's around 1k opened issues on github. If you want more wide adoption you can help with fixing them.

The Python world is small, and a lot depends on being in the right cliques and working for the right corporations, regardless of the quality (or even quantity!) of output.

Conferences are regulated with always the same speakers who have no incentives to change their slide decks. Advising people about the broken packaging system is also a good source of income.

As someone else has said here, it's an authoritarian club.

There's a bigger problem. Comes down to production issues and the efficiency of using something just because it's cool. I'm a grizzled setuptools guy, for sure. While I think poetry has many great ideas, it currently can't be used in a container-based microservices ecosystem with even one problematic dependency (without monkeypatching or forking crazy things like `requests` in some cases). Proprietary dependencies make it even more painful, but that can be fixed.

My current organization had adopted poetry before I arrived, and I eventually had to decide that we're reversing from poetry, back to pip and explicit usage of setuptools. It was too problematic, and was costing us in production DLL-hell bugs. The solver does not allow you to pin a constraint if it conflicts with the packaging metadata of your dependencies.

The stance of the poetry team is that any discrepancies should be fixed in upstream dependency. Hey, I love purity and platonic solids too, but... so that's not a completely crazy design decision, but we're in an NP problem here. That stance requires the cooperation of every single upstream package maintainer, which obviously, no one can actually operationalize. Here is a very old issue about this:

https://github.com/python-poetry/poetry/issues/697#issuecomm...

There is some movement, and acknowledgment that a tool is useless if you can't take it full-stack for your application. As of writing, the state of the art is to fork and patch things like fastapi and flask. That's nuts. I can make our stack work with the tested tools much more consistently, and there isn't much downside.

https://github.com/python-poetry/poetry/issues/697#issuecomm...

To me, that removes the value of using poetry. `poetry install ` is something you do very rarely. Propping up new python environments needs to happen very frequently. The convenience doesn't outweigh the operational costs.

tl;dr: I think poetry is really cool, and it looks like npm to a developer. It doesn't play nice with others, and that's by design. For production use, pip/setuptools solves more problems than people think it does. Sure, constraints.txt is a blunderbuss for a dependency resolver. However, it works. Especially if you are making more than one package for a real-world application. If you are maintaining a standalone library without complex dependencies, sure, go for it :)

Poetry just was too little too late, containerization is how people are fixing their dependency issues now. I obviously want have a dockerfile that really represents all my dependency then a good package manager that blames my OS when it can't install something. It's the best horse at the car show.

Imho the python steering committee is too weak in taking action in the direction of improving the whole ecosystem.

They did mess up the python 2/3 and imho they just don't want to deal with such things anymore.

Deprecating stuff is necessary as well as it is necessary to provide an upgrade path (both documentation and tooling) and new stuff.