I'm tired creating backups, or loosing data. Let's be honest, sometimes it happens. Does a filesystem that never deletes/overwrites anything exist?
I envision something like an append only filesystem where you all files in your hard drive are under some OS level 'version control' and you can rollback to any version you want. Only files in my home dir would be enough. Obviously once space runs out it would start overwriting the oldest versions.
The affordable version of what you are describing is a set of NAS class servers that use something like DRBD or Ceph for multi-node replication and rsnapshot for local snapshots/versioning if LVM. If using LVM, then rsnapshot should be configured to save its snapshot where the remote clients can not write to it.
Look into setting up Ceph clusters if you plan to build something that will need to scale really large on individual volumes across multiple nodes. Ceph supports creating snapshots. There are pros and cons to solutions such as DRBD, LVM and Ceph. That would be a topic in and of itself. Others here are mentioning ZFS and that is also a popular solution but I have never used it and can't comment on it.
In summary if I was asked to build something that would grow to unknown size, require snapshots to roll-back files to a specific date and be fault tolerant I would go with Ceph. As a bonus feature there are libraries and tools to present Ceph volumes as S3-like buckets. There are ansible playbooks for setting up Ceph clusters. Ceph has some really good security features as well.
> Obviously once space runs out it would start overwriting the oldest versions.
You can’t have both.
Also, your system wouldn’t prevent data loss on hardware failures.
What you describe is/may be a WORM device (https://en.wikipedia.org/wiki/Write_once_read_many)
In practice we used to work on things then purge when happy with the final changes, as we didn't have that much 'spare' disk, but it was a nice feature.
OpenVMS also has a versioning filesystem and a NFS server. The x64 version is available as a private beta, but it's not cheap.
Personally, I just keep most stuff in a Google Drive folder and important stuff gets the keep versions forever bit turned on. Everything else goes in a git tree and have some automation to clone it to a reliable utility server when the OS thinks there's plenty of bandwidth available.
It's worth noting that the Google Drive solution is the only one that I've been able to get to work reliably across Linux, macOS, Windows, Android, and iOS. I've tried OneDrive, Dropbox, and iCloud Files, but they all have various problems with conflict resolution, annoying upselling, or limited file type support for versioning.
A list of options from a report:
https://www.fsl.cs.sunysb.edu/docs/secfs-rpe/index.html#tth_...
VersionFS:
https://www.fsl.cs.sunysb.edu/docs/versionfs-msthesis/index....
Elephant:
https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.70....
https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.129...
Silver:
https://www.usenix.org/system/files/conference/hotstorage16/...
Also possibly of interest, vfs:
I know that overlayfs exists, which could enable this sort of thing by capturing each individual change to the filesystem as a new layer. https://www.kernel.org/doc/html/latest/filesystems/overlayfs...
Btrfs and ZFS can do snapshotting, and I would imagine it could be configured to snapshot at certain intervals or after individual changes to the filesystem or a section of the filesystem, like watched directories.
https://wiki.archlinux.org/title/Btrfs#Snapshots
https://docs.oracle.com/cd/E19253-01/819-5461/gbcya/index.ht...
There's also filesystem-in-userland tech like Fuse which could enable something like this. https://www.kernel.org/doc/html/latest/filesystems/fuse.html
On macOS, there is Time Machine built in. https://support.apple.com/en-us/HT201250
ZFS does have the concept of Snapshots but I would not consider that user friendly or useful for version tracking.
There is plenty of Version Control systems out there, like Git, again not user friendly for what you want.
Dropbox/OneDrive/Google Drive provide various levels of version tracking of changes.
I personally use Backblaze (30 days, continuous) and Tarsnap (45 days, nightly) for both off-site and temporal recovery when I really need it. Backblaze is used for "desktop" systems, Tarsnap for "server" systems.
* use ZFS (or any snapshot capable file system)
* trigger snapshots at regular (short) intervals (as chosen by you).
ZFS snapshots are very low cost, and you can browse them using your normal file tools. You could reasonably snapshot every (say) 5 minutes, but I wouldn't try every second :)
I use linux and system link the home folder to a Dropbox folder. There is an extended history option [1], so that you can go back a year for any file. I think it’s a reasonably close solution if you do not mind the cost and syncing to cloud.
[1]https://help.dropbox.com/files-folders/restore-delete/extend...
Although the website seems to have not been updated in a while, it's mailing lists appear to be active. [1]
https://nilfs.sourceforge.io/en/ [0] https://marc.info/?l=linux-nilfs [1]
I also desire an easy solution to this usecase.
I'm tired creating backups
Have you tried to automate them? Thousands of free and paid software to do that. Setup once, forget until next loss.
More seriously, in a time of unobtrusive automated backups and dirt cheap storage space, why is this such an issue where you want to have this kind of system?