Why Python?

In my and many other developers experience and as indicated by studies, the expected number of bugs per line of code is constant and independent of the programming language. Thus, say, 400 lines of Python will contain the same number of bugs as 400 lines of C. But 400 lines of Python contains much more functionality than 400 lines of C. This is what you have been missing.

>Like I can use Node, Go, C#, and Java at scale without being concerned about the low-level details, however, that's not the case for Python.

Well, if you chose not be concerned about the low-level details with these languages, then you can chose not to be concerned about the low-level details with Python. There's nothing inherent in Python or those other languages that makes it otherwise.

Or you mean "but Python is slow"? That's mostly irrelevant for many use cases, which either don't need the speed (web, administration, glue, controllers, etc), or depend on Python packages written in C/Fortran and plenty fast (which is the case for AI / ML / DS and such).

>it doesn't seem to offer any benefits compared to Node, Go, C# or even Java

Ease of use, less ceremony, huge package ecosystem including several de-facto standards for many use cases/industries, suitable for glue work, are benefits.

Python is easy to use and easy to start programming in. Much easier than the other alternatives you mention. This from a guy who uses python but does not particularly care for it. I love me some curly braces and hate significant whitespace but there is no getting around the ease of teaching in python and its beginner friendliness.

Why cant you use python at scale? Many ML/AI/Data tasks are exceedingly resource intensive but python still plays in thoses spaces as you elude to.

Are people writing realtime systems in python? Probably not but dont be quick to dismiss python before you know it is not performant enough for your use case.

I'm not saying python is the best choice, but it is often the easiest choice.

Author here! I have finally come to realize why and where Python is required.

- Python is a fantastic frontend (glue) language that interop well with C/C++ libraries. This seamless binding is not natively available in languages in Go or Java or even JS.

- Python has an excellent data science / modelling / data engineering / ml ecosystem. There is no other language that comes close.

- Python is a fantastic MVP language. One can rapidly prototype and iterate and to a very good extend scale a product.

Python was favored by universities, professors and the like a long time ago. Maybe it just had this academic feel to it.

This is even why it started to pick up in AI / ML / etc space. Some students that used it during class may also carry on with it.

It was also common for sysadmins before Go took over. A lot of the earlier tools were Python based.

Usually languages are created by people and because people like it.

You can't use Java at scale without worrying about the low level details (e.g. performance). So in a sense python is in the same club as java.

I don't understand the need for python either. I've built data processing and visualization tools in C++ and prefer to do that so I am aware of the performance implications. With python you still need to be aware of what happens under the hood, but then you also need to learn the syntax and identation, so it's like you have multiple jobs. Why not learn C or C++ and use that? There are endless libraries you can use. AFAIK anything that exists for python can be used from C or C++.

It's especially puzzling when getting to a coding interview and the person only knows python and doesn't understand why you never had the need for pandas or any other python specific libraries.

I'm wondering if python hasn't been pushed such that more coder drones can be created easily which makes it easier to hire people and pay them less.

There are advantages working in an interpretive (scripting) language and there are several science applications that use Python as the top level control language and a c-like language as the heavy lifting code. This plays to the strength of each language and casual users can use the complex functionality without looking at the complex code.

I got into Python from C/C++ for data plumbing in the late 2000s, before data science / ML really took off.

I think much of what appeals to me comes in three camps:

One: "Zen of Python" philosophy. The idea of both explicitness and being "exactly one way of doing things" helps with menial tasks I don't want to think too deeply about. Maybe I _shouldnt_ think too deeply about how I move some files around, parse JSON, or hit an API.

Two: Batteries included. The standard library has always been extensive and "just works" in ways other language ecosystems have long struggled to catch up to. Especially for data / file plumbing tasks. It's also not very complicated to do most simple things in without a 3rd party library.

Three: C integration. Writing native C modules seems to have been baked into the language from day one which has led IMO to much of the data science ecosystem. The ecosystem seemed to connect with the practical, old-school C programmer part of my brain, and make a lot of sense from a perspective of "get shit done" imperative programming style.

One maybe critique -

I think the nature of Python, and how people historically approached it, can lend to a kind of lazy/bad software engineering.

People historically had gotten into Python because of dumb scripts, and not because they care to think deeply about how code SHOULD be structured. Python's practicality is a bit in contrast with the culture of Ruby where aesthetics are highly prized. So a criticism of Python could be its very easy to get started with a dumb script, to solve a practical problem, that eventually becomes a critically important project. Unfortunately everyone just focusing on "getting the next thing done" means it can morph into unmaintainable spaghetti mess because its some side thing.

I don't know if that's the languages fault per-se, or just a side effect of the "lack of ceremony" needed to do work. Certainly, you can and should have high software engineering standards for your Python project. Just something I've observed perhaps as Python itself can often be a "side" thing, not the primary focus of practitioners.

ecosystem + simplicity. performance matters once the time to write code that works is less than the time it runs for. and then once that gets surpassed, port the hot path into c/c++, boom.

Thanks to Numba, you can even JIT python to native machine code and gain 10 - 30x performance boost. And you can even compile the code to CUDA GPU code! Pretty cool to do that in python, rather than in a C dialect.

> however, that's not the case for Python

in what way?

You bring up a valid question. The value of Python as a general-purpose programming language comes from a variety of its attributes.

Readability and Simplicity: Python was designed to emphasize readability and simplicity, making it an excellent language for beginners and for prototyping complex systems. This simplicity means developers can spend less time understanding the code and more time creating functional elements. Even experienced programmers appreciate Python for its simplicity and how it encourages good programming style.

Wide Range of Libraries: While you mentioned AI/ML/Data, it's worth noting that Python's rich set of libraries (both built-in and third-party) makes it useful for a massive array of tasks beyond just those fields. It has powerful libraries for web development (Django, Flask), data manipulation (Pandas), image processing (PIL, OpenCV), GUI development (Tkinter, PyQt), and many others. This versatility allows Python to be used in many different problem domains.

Glue Language: Python is also often used as a "glue" language to connect different components of an application. Since it's easy to interface with C/C++, it can be used to control and orchestrate low-level modules while providing a high-level interface to the user.

Automation and Scripting: Python's simplicity and wide range of libraries make it a popular choice for automating tasks, which is a large part of what any scaled system needs to handle. These tasks range from managing file and directory operations to web scraping or interacting with network resources.

Testing: Python is often used in the development environment for testing other code. Its simplicity allows for writing tests quickly, and its wide range of libraries allows for testing various programs.

Education and Research: Python is also widely used in education and research due to its ease of learning and wide range of applications. As these researchers move into industry, they often continue to use the tools they are most comfortable with.

In terms of using Python at scale, Python's indeed interpreted nature and Global Interpreter Lock (GIL) can limit its performance in some scenarios compared to compiled languages. However, many organizations find that Python's benefits outweigh these costs, especially when the limitations can be mitigated. For instance, using Python as a high-level interface to low-level modules written in C/C++, or using multiprocessing to take advantage of multiple cores and systems.

So while it's true that Python might not be the best choice for every situation (for instance, real-time systems, high-performance computing, or mobile app development), it's a powerful tool with a wide range of uses that many organizations find valuable. Different programming languages often have different strengths and weaknesses, and the best language to use often depends on the specific needs and constraints of the project.