HACKER Q&A
📣 jzombie

Pip, Poetry, etc. for Python package management?


I was personally using poetry to manage packages, as I thought it was easier to set up than pip, but never liked the fact that poetry doesn't use pip under the hood, as pip just seems more standard.

So, my question is, what do you prefer, and why?


  👤 ActorNightly Accepted Answer ✓
Pip.

Generally, reliable python setup is really easy (at least on linux, don't know about other platforms).

1. Don't use the prepackaged system python. You don't want package installs interfering with stuff your distro needs. Build python from scratch from source after apt-get or pacman installing the necessary dependencies for full functionality, then use make altinstall. It will leave the binaries in the major.minor version format without overwriting the python3 symlink.

2. Use get-pip.py to install pip by invoking it directly with the python version you just built. This will alias pip and pip3 to point to the correct version. Optionally, you can uninstall any package manager installed pip for safety. The pip executables should be in .local/bin

3. Never run pip as sudo, so that it install packages in user home directory (.local/lib)

4. If you need version control, use virtual environments.


👤 saasjosh
I use Poetry because it makes my life easy and works just like Ruby's bundler and npm in Nodejs.

Some people hate it because they forced people to upgrade a while ago. It's childish to dismiss such an amazing tool because of one bad decision on the maintainers part.


👤 bnchrch
Love this question as it highlights pythons largest shortcomming:

Out of all the modern languages they have the worst dependency story by a mile.

Most of the issues with python dependencies stem from:

- Global package installation is the default behaviour

- Libraries sub dependencies must be compatible with other installed library subdependencies

- Very loose and permissive version requirements as a result

- Leading to sub dependencies breaking your program between builds

- No native support for lock files to prevent this

Poetry so far has been the best solution to this systemic problem.


👤 cdaringe
Pipenv. https://pipenv.pypa.io/en/latest/

I use VENV_IN_PROJECT=1 too.

End result is reproducible installs and easily inspectable dependencies and interpreter right next to my source.

Closest thing to cargo or npm ive found.


👤 PaulHoule
Pip doesn’t really work. That is, it works sometimes, but instead of getting a global view of the different versions of dependencies, it just starts downloading and installing things and it can get itself stuck in a corner.

Some developers use pip all the time. I have some little projects that I do in PyCharm and I add packages with the IDE which uses pip and it is fine.

I worked at a place where our projects were pretty complicated and pip could not solve them consistently.

My RSS reader YOShInOn uses conda for package management because CUDA “just works” with conda plus you get numpy and the like compiled against the MKL libraries for a nice boost. Conda is slower than it should be, I think it was a serious mistake that they compress packages with BZ2 (I had a system that even installed a local copy of all the CUDA libs with conda packages I made but decompression just takes forever.). Still, the solver is logically sound.

The worst problem of it is that the packages you want might not be in conda, you can get away with pipping in a package or two but long term that’s the road to ruin. If you do want to use packages that aren’t in conda your beat way forward is to build conda packages for them which is usually not hard —- to really get control of the build of any egg packages you need to make wheels out of them anyway which is basically the same.

It is a little ugly but YOShInon has a sidecar application which doesn’t have any machine learning in it but instead links the mastodon API client and other things that aren’t in conda so YOShInOn just runs that executable when it wants to post to Mastodon.


👤 8b16380d
pip and virtualenv because I have always used it and have never have run into issues at scale.

👤 catlover76
I like PDM, personally. It's like Poetry, but feels a bit more ergonomic, supports a means of storing deps other than virtualenvs (a rejected PEP submission that basically works like node_modules), and never did that BS Poetry did where it failed peoples' builds until they upgraded or w/e

Perfunctory note: Python is garbage, in part because of its "devops". Also, the ecosystem seems like it's full of trash coding standards. Also, it's slow. I can't believe we are all continuing to use it :smiley_tear: