HACKER Q&A
📣 whinvik

Why do Python libraries make so many breaking changes?


Today I was working with Pydantic and I realized that I am version 2, and so many things are not the same anymore. Of course, it's good that they have put warnings in there but I still don't like it.

I am also a heavy user of Poetry and I am in dread of updating it. Everytime an update happens, something breaks. Do other people have the same experience? Is this a Python only thing?


  👤 scolvin Accepted Answer ✓
Author of Pydantic here, very sorry you've had problems migrating. We were very aware this was going to cause problems, and have done our best to make migrating easier. I truly believe that v2 will be a big improvement.

I think there are two things going on:

1. Python: Python's laxness doesn't make it easy to define strict APIs or know when you're breaking them. The lack of export controls also means defining what's in the API and what's not is hard. Also Python's strong introspection makes it possible to do crazy things further blurring the API boundary.

2. The ecosystem: with pydantic as an example - the library has grown enormously, more than I ever imagined from an experiment in a file in another project to a major library used by thousands of millions of developers, for a very wise range of applications. I never imagined that happening, do I didn't think all that hard about the interface at the beginning, or even at the v1 release. I also knew much less back then.

I'm not sure this kind of growth is very common in other languages except JavaScript.

(Written on my phone, sorry for typos)


👤 smokel
This happens all over the software world. The same goes for the JavaScript ecosystem, and even though it's slightly less bad in Java land, it happens there too. I personally think that there are both social and technological reasons for this.

Java developers, for example, tend to be slightly older, and therefore have had more painful experiences upgrading stuff, but they also tend to work in large enterprises where breaking stuff has an associated cost, that you do not experience when you are hacking on your own. Not breaking stuff requires an insane amount of energy and discipline.

For the technological argument, I like to think back to the 1990s, when I experimented with Common Lisp, and was amazed at how long one could work on one piece of software, without having to restart any of it. Oftentimes, the runtime state and the source code on disk would get terribly out of sync. It was even a best practice to simply store the memory image to disk, and restart from that later. I envisioned a future in which these problems would be solved, and instead of writing new programs, we would merely polish and improve running software, keeping the data alive for eternity.

Unfortunately, it's now 30 years later, and the only movement in that direction I see is that data remains somewhat active while being shared by multiple services. Unfortunately, the services themselves are still blobs that must be restarted over and over again, losing all intermediate state.

I think we could do better, for example by having a programming language that takes immutability to the next level, i.e. by not allowing a restart, or by requiring upgrade functions to go from old data formats to newer ones. Perhaps something may have come out of the field of reversible programming languages?


👤 jjgreen
Python is notorious for this and has been for years. I'm afraid you just have to accept it as the price for an easy syntax and access to lots of nice libraries.