Alternatives to organizing code in files and folders?
I realize I spend a lot of time organizing code into files and folders. And usually, it's organizing packages, modules, functions and classes into matching files/folders, and then updating package/module import statements. That is, it's simply a chore.
A barrier to change is our current tools such as IDEs and VCS/Github are tightly tied to the files and folders paradigm.
Though, every tool that doesn't map 1-1 with the filesystem I have always hated the experience. Think: XCode, Visual Studio. But maybe it just hasn't been done right.
What are your thoughts on alternatives? Drawbacks to change?
I actually like the filesystem and its hierarchical structure. What is missing is speedy access via alternative ways similar to views in SQL where you can narrow down your criteria based on filenames, dates, and contents. Finding something in a huge collection of source files used to be very slow until I discovered ripgrep which is insanely fast.
Hierarchical structure has been criticized a lot in recent years, and rightly so as for the naive view which insists that there is One Right Way to categorize things. But the not-so naive view acknowledges that any hierarchy is artificial and ultimately arbitrary, while it at the same time asserts that it can also be useful. Think of a library: which one is more useful to your most frequent use cases, one that keeps books ordered in divisions and subdivisions of subjects on the shelves, or one that keeps books in random order? I think it is the first. You'll still need one or more catalogs and one can still debate where that book on maths for the engineer goes (maths, engineering, or text books); that doesn't mean the hierarchy of the books on the shelves is useless.
A huge drawback to changing this is all of our computing resources are based on a 100+ year old concept of a file cabinet with folders and files. It has a lot of historical traction which you would be attempting to go up against with a different concept.
As long as the code is easy to find and doesn't have similar naming as other code you should be good to organize however you want. If you work on a team then make some decisions as a team on how to organize files. I work with Rails which has an opinionated view on folder/file structure which is great because I don't have to spend time thinking about it and can focus on getting things done.
Revamp an IDE to abstract away from the file system. Let the IDE organize the code according to functions and subsystems instead of files. Add tools to the IDE for better search, and the ability to hide sections of code and comments when you don't want to see them. Add an automatic map builder that outlines control flow and indicates which sections of code are closely related. Add an AI to suggest code and to analyze and summarize the structure of the code. The code all gets compiled into one binary or app so it is logical to treat all of it as one monolithic thing, but people aren't capable of even seeing 300 pages of code at the same time, they can only keep small sections visible and work with them at one time. Design the IDE to handle that complexity and make it easy to zoom in on the code that is relevant to what you want to do.
Smalltalk. And yes, mapping code components to filesystem objects is a kind of an impedance mismatch.
For the current reality that is directories and files, you can get pretty far with sed, grep, git grep, shell scripting, find -type f -name 'foo' -exec ... '{}' ';', etc, etc...
If some alternate reality existed, based on some other data structure / schema... I guess some other set of tools and workflows would be needed to find and move whatever the new name for blobs of bytes (files) would be...
As an experiment try putting all the code in a single file (I am not recommending this). It then becomes impossible to hide complexity without decreasing it, and you naturaly come up with all sorts of utility functions. The code becomes denser.
It is an interesting exercise, and with a good typesystem it can go a long way.
What if you had one giant folder with every file available? The obvious problem is polluting the namespace with multiple same named files, naming things and enforcing UUID’s becomes annoying.
I think folders and sub folders makes sense for run times and IDE’s to “know where to look”.
The file system works and is a common denominator. Even if you could come up with a better scheme for organizing software projects, what use would it be if no tools except for your own supports it? Using ls for listing modules/classes in a project is useful, as is removing such items with rm. So is grepping, or using some other editor than your preferred one to interact with the project. Maybe something better will show up in the future - like asking ChatGPT where your stuff is - but I'm skeptical.
Hierarchies (folders) and tags are the only way to organize things.
I’ve been curious about this topic for a while primarily because I’ve found that issues with circular dependencies/imports in my projects are simply a consequence of their organization into files.
If every exported variable or function was in a separate file, that would eliminate most of my problems. But it’s just a pain to write code that way…
Is the filesystem interface itself the pain point? Would you rather be organizing URLs? Or is it the compound nature of source versus build machinery? Would you rather lodge your source and build machinery in polyglot?
Pycharm help you to refactor variables/functions/class as long as they have unique names in the first place. Otherwise everything messed up together.
They do it by auto renaming and something else.