How do you manage your “family data warehouse”?

Safe deposit box at the bank for the physical records. A labeled manila envelope stored locally for each of us with the typical "identity" documents, SSN card, birth certificate, etc.

Everything is stored locally, unencrypted, and backed up on a local removable HDD, and on BackBlaze. BackBlaze has saved my bacon more than once.

As for physical objects, they're all temporarily mine, until I pass on. They're either tools, consumables, or tied to personal memories. Everything else can go.

Documents are in Subversion hosted on a local home Debian server which is also NAS.

Photos (hundreds of GBs) are also in Subversion hosted on same server. Photos are managed/viewed either via Digikam (on both Mac, Linux, Windows) as well as through a generated HTML/js gallery served on the same server.

Home server is backed up via restic to

* local harddrives that are rotated weekly and moved offsite. * to a rented physical Debian server in a different country * to a cloud storage provider

Server storage is fully encrypted using LUKS. Backups are encrypted via restic/ssh.

Various sensitive documents are separately encrypted inside the svn repos via 'age' or similar tools.

This setup has been working for almost two decades. I have yet to lose any data.

Fairly simple here. Everything goes in iCloud with offline backups via Time machine. Documents are scanned and disposed of where possible - collecting paper is a bad bad bad thing. Identity and required paper documents are kept in a locked drawer. Credentials are stored in Keepass. Photos and videos are all in Apple Photos. Instructions are left on how to deal with the offline backup should anyone need it.

Obligation wise it's good to keep all documents for whatever the longest record keeping period is in your region.

Important that you scan your identity documents and keep copies too. That has been incredibly useful when I'm abroad because a lot of people will retain passports etc if you go in a hospital or something so you can just show them the electronic copy and tell them you left it in the hotel.

All types of documents go into a paperless ngx. All documents are tagged properly so that I can find them again. Also the search worked quite well so far. I’m still archiving the previous years currently, as I set up everything this year.

Backups via restic end up on backblaze. Haven’t gotten around to set up the local backup. This is a good reminder to do so. :)

The photos end up in iCloud and Adobe with a backup in backblaze. Not so happy yet with this solution. But most alternatives I tried weren’t as comfortable.

Edit: The paperless ngx server is a low power machine that runs in my local network.

I live in a paperful country, and I have no scanner.

Important stuff has a photo taken, and it's likely backed up in my photo archives (Syncthing → Pi Storage → NAS backup).

My girlfriend still uses google and whatsapp, and I tend to send her important media in our chats, so we'd both have to be kicked out of our gmail and whatsapp accounts to truly lose everything there.

In Belgium, every citizen has access to a "digital vault" provided by the government, which is accessible using their ID card. Individuals can upload up to 1GB of documents into this vault. It's a well-designed system that ensures privacy during one's lifetime, but you can choose what happens with it after your passing.

https://www.izimi.be/en/

Dropbox for everything except photos, videos and security-related info.

I used to keep photos and videos on an external hard disk with a jwz-style backup mechanism and Lightroom 3 for organizing it (kept the light room database file on the hard disk itself)

Now just gave up and use iCloud Photos. Not as good as Lightroom but can't find anything better that's also easy.

Secure details go into 1Password with a printout of emergency keys in a wardrobe under lock.

I used to use Evernote but moved away from it soon after they started introducing chat and a bunch of crap no one cared about.

I keep copies of often-referenced items from Dropbox in Apple Notes but the master copy is a file in Dropbox.

I'm using the Shortcuts app on the phone to take a receipt from an online payment and file it away in the /receipts folder in Dropbox.

I used to have an IFTTT integration to take all the photos I posted on Facebook and keep a copy of them in a Dropbox folder as well. It probably still works but I don't really post stuff anywhere any more.

I have a messy drawer where I dump all of this stuff. It’s basically write-only, once I throw something in there I know I’ll never look for it again.

For digital stuff it’s mostly the same idea. If it’s email (which is almost always the case), I archive and forget about it. If it’s something else, I’ll throw it in my virtual “messy drawer” which is just a folder in Google Drive.

You are getting quite a few clever ideas in this thread, but I suspect the vast majority of people out there in the real world do more or less the same as I do (of course, I’m biased).

My “philosophy” (overstatement) is that 99.99% of the time I’ll never need it. If I ever do, it might be a little painful to dig around, but that doesn’t justify having a whole system in place. I would rather keep my mess.

A Fuji Snapscan S1500 + folder of PDFs on my laptop + full text search. Mylio for photos. I sometimes purge and file, but not very often.

There's always a bit too much that is attached to emails, I try to remember to save stuff out.

A Synology NAS in the attic which backs up the above.

No encryption.

An £50 "fireproof" box for passports, citizenship certs, things where the original is important.

I assembled a ridiculous old server to do tape backups but couldn't make a very practical system - I wish I could attach an LTO drive to something smaller!

Fortunately for me, most of the official documents here are digitized, and stored on government servers, so in the event of my death, my family can request access to relevant documents. That takes care of most of the official stuff. Also, everything legal i cosigned by my wife, so either one of us has equal access to it, but like official documents, these documents are also stored in government databases.

As for products, i have aimed to buy services that are easily transferrable, i.e. we use iCloud family sharing, meaning every user has their own storage, but they piggy back on my paid subscription. If/when i stop paying, they can chose to continue paying by themselves, and things will just "magically" keep working, but otherwise it's a tricky game.

As for memories, i store everything in the cloud. Most is encrypted by Cryptomator. I have a small server at home synchronizing data home in real time, and the server backs up data hourly to a local target, and a couple of times per day to another cloud.

I have thought long and hard about decentralizing this to let each users laptop do this, but with a family photo album of 3.5TB that would mean i lose deduplication.

It would also put much more demand on the client hardware, as it would need to be able to hold the entire photo library to back it up. I have LONG REQUESTED a way to backup iCloud data without the need to keep it all local, i.e. download upon use for backup, and delete again, but sadly that's not how it works. If your data is "cloud only", your backup will only backup file pointers, and not the actual photos.

Once every year i make Blu-Ray M-disc archives of the previous years photos, and to some lesser extent also documents. I make identical copies and store them in geographically separate locations. I use no encryption or compression on these archive discs. I also have a set of identical external USB drives that i update with the entire archive yearly and then rotate between locations.

I am fully aware that optical media may not be around for much longer, but that's a problem for future me. For now it is the best and most affordable "long term, no maintenance" storage technology available to consumers. Until an official "end of life" is reported, i will keep using it. Once it's EOL i can easily migrate my archive to "something more better", even if that ends up being just ordering photo copies.

I'm a bit shocked that all answers so far use cloud storage and not a single one proposes offline.

I store everything in the cloud, because it is unbeatably convenient except for the very things mentioned: health, financial and legal. I'm not 100% strict either, because if a document has been sent or received electronically or can be assumed to be stored in the cloud anyways I really can't do much. I still assume that every online storage will be breached one day and keep some documents deliberately offline.

To give you some chills, check out "Who's been pwned" [1] and this is only the tip of the iceberg that has surfaced and that fulfills HIBP's inclusion criteria.

[1] https://haveibeenpwned.com/PwnedWebsites

Apple solves this problem for me entirely. No need to tag documents etc, just iCloud everything -> everything is searchable.

Having built complicated storage systems during work hours I had a strong urge to build a home NAS. What stops me is that I have this Documents directory on my laptop, which was salvaged from another disk that failed about three years ago, and served for 11 years prior to that, and some documents were copied to it from a PC I owned before then... and going farther back doesn't really make sense as none of the documents would be really relevant / authorities at the time couldn't deal with digital copies of documents, so, storing them made no sense.

So, in the end, I just keep them in the Documents directory. Some, most commonly used are also in my email. That's it.

I've been doing the data storage tango for thirty years now, and the one constant is that everything requires maintenance or has to be moved to a different platform and software eventually. Today, everything is in Google Drive and Google Photos; about 600 GB worth of data and media. Maintenance consists of an annual Takeout ordeal with copies going on two different removable drives, one of which is stored in a so-called fireproof bag.

The fireproof bag also has my account/password list (printed out once a year), birth certificates, titles, and passports.

If you are the tech person in your family, think about what happens if you die. Will your family be able to figure out and maintain your system? Mine is far from perfect, but most of my family can figure out Google Drive.

A Synology NAS running Portainer (https://www.portainer.io/) running Paperless NGX (https://github.com/paperless-ngx/paperless-ngx)

This works better than I can possibly tell you.

I have an Epson WorkForce ES-580W that I bought when my mother passed away to bulk scan documents and it scans everything, double-sided if required, multi-page PDFs if required, at very high speed and uploads everything to OneDrive, at which point I drag and drop everything into Paperless.

I could, thinking about it, have the scanner email stuff to Paperless. Might investigate that today.

Paperless will OCR it and make it all searchable. This setup is amazing, I love living in the future.

We scan every document and put it in a folder synced with SyncThing. We throw the original away unless it is something official like a birth certificate.

We use SyncThing because it is free, doesn't require user accounts, and works across Mac, Windows, Linux, iOS (Möbius) and Android.

We have a bunch of high level folders (eg. Health, School, etc). Inside that folder every document is named starting with a date, eg: "2023-09-11 Invoice Dentist Kid1.pdf".

So far I've always found every document.

TrueNAS Scale (formerly FreeNAS) server with 6 x 4TB ZFS storage in a Raidz2 (or was it z3?) configuration.

I rsync my picture and document library from my MBP to the NAS. The MBP also has a Backblaze backup. 2-4 times a year I sync my pics, docs, and backups to a 6TB drive that’s offline.

Laptop documents folder and outlook inbox

OneDrive and a fire resistant box. I suppose I actually store far more in my emails than I should!

A few folks have talked about digitizing paper documents, so one of my favorite tricks for iPhone havers:

iPhones have very nice scanning tools built in, but it’s buried in the Notes app. Create a new note, click the camera icon, then “scan documents” and it will create a very nice, usually well cropped and OCRd scan that’s saved as a PDF you can then export elsewhere. Wish it was a standalone app because it works so well, and this is from someone who helps digitize and preserve paper documents for a living.

An organised chaos of diverse storage. On a NAS, Google Docs, Dropbox, Flickr, variuos photo apps now offline (coppermine etc). All newer photos are on icloud though also backed up offline. A lot of older paper documents in physical folders, some organised but most just bungled together. Old physical photos are hopefully somewhere in the attic.

Most recent years letters are however photographed on a phone and put into relevant shared photoalbums, then shredded. Albums by year and category e.g. "School docs", "Kids School work", bills, statements, manuals and instructions, etc. Still have a big chore to go through all the old folders to scan and shred so we can really be paperless.

A few times I have had to dig out old statements or proof of address on a recent bill. However they all just wanted a photo of it, so storing it digitally would have been fine.

Really important documents are paper and stored in a filing cabinet into labeled folders. Every so often we go through these and discard the papers we no longer need (old tax records beyond 7 years, utility bills more than 6 months old, etc). If our house burns down or floods badly, we'll potentially lose these documents, and that's a risk we're willing to accept. Pretty much all of these documents are actually replaceable with effort, time, and a little money. I can get new notarized copies of birth certificates, old tax documents, will, passports, etc if I really need to but it would be quite annoying. The risk of losing them and the annoyance even each other out, it's a very small risk. If I die, no one needs to know how to use a computer in order to access my will, taxes, etc.

Important digital files are all on our family computer which is backed up to a cloud service which will mail me a disk to do a restore if needed. This includes all our photos and videos from since we've had digital cameras. If both my wife and I both die, no one will have any clue how to deal with this and that's OK.

Photos we really like are printed and in photo albums organized by date. We'll lose these in a house fire and that's a risk we're willing to accept. The ability to easily pull a physical album and see pictures from a past time grouped together and curated is something we really like.

I use https://www.trustworthy.com/ and Dropbox and am very happy with this combination.

I scan all documents and paper mail and store the scans in a git annex repo with a few replicas on machines I own or rent. Originals go into binders, ordered by the hash of the document. With a few helper scripts and OCR with Tesseract searching through my documents is surprisingly fast, so is adding new documents at the proper place in the binders. The idea behind ordering by sha1 of the scan was that the binders would be storage only and all categorizing and organizing should be digital, and also that as I add more documents load balancing between the binders is trivial: at the moment I have one binder for hash prefixes 0-7 and one for 8-f, with a divider for each one-hex-digit prefix. If I ever feel there are too many docs per divider I can split further.

Emails go into a plain git repo, and photos go into another git annex repo.

Ideally I’d like to put all of that into Perkeep because I like the idea of just throwing anonymous blobs into a bucket with the possibility to organize and tag them after the fact, but it looks like that project is dead and I would need to add a few things to it to support my use cases.

I store a lot of scans and emails I won’t ever need again. I just consider that storage becomes cheaper faster than my storage needs increase, and that storing extra documents and saving myself the need to decide whether it’s worth keeping, is a good tradeoff.

Apple solves this problem well! Note that I use iCloud family which allows me to share the storage with my family. But for going down the list:

A) Mint for most financials. iCloud for storage of documents.

B) Apple Health for keeping track of my records. For ad-hoc stuff I use iCloud to store the documents.

C) Product and service info I use iCloud for. Logins and credentials to them are stored in my family's 1Password vault.

D) All legal documents stored on iCloud.

E) For photos and videos I use iCloud Shared Albums. It's unlimited free online storage of photos. Otherwise it's a mix of iCloud storage and the Notes app on Apple devices.

Obviously you can tell I'm deep in the iCloud ecosystem but personally I feel it's worth it. The synchronization and ease of use for my family is well worth it. For what it's worth I'm on the 200GB plan.

Google photos for photos and videos, I use storage saver which reduces quality but whatever.

Everything else goes into Google drive. I use the Google drive app on Android to scan my physical documents, and I pay for the 200GB plan which is enough for all my stuff.

I've been hoping to write a script to backup stuff into an S3 bucket. I just never have the time.

I separate my data into two categories, fast (and small) and slow (but big). Both types are important, but separation makes management much easier.

Fast data:

    - documents, pdfs, config files, notes txt etc.
    - these files are synced directly, on-site and off-site

Slow data:

    - archives, images, music, large backups etc.
    - these files are transfered to off-site archive on a per-need basis
    - they are made available in the (family) cloud as read-only
    - they are made available to selected internal services read-only (music service, photo service etc.)

Tech stack:

    - I self-host
    - I follow best practices (zfs, 3-2-1 rule, vpn-only)
    - currently I use Nextcloud, rsync and borgmatic

A folder on my desktop named “stuff”.

On a more serious note I use MS OneDrive family account and iCloud - I figure if either of those companies goes under there are bigger things to worry about. I print a lot of my photos so I’ve always got hard copies of the kids when they were growing up - everything else is unimportant or replaceable.

If you're doing all the data archiving yourself, and only need to make it readable to other family members...

I have a filesystem tree that organizes digital copies of all financial documents/data.

The tree is routinely backed up to rugged USB flash drives, labelmakered "FOR HEIRS", and stored in my safe and bank safe deposit box. The drives are unencrypted, and in FAT32 format (though I use Linux), to increase the likelihood that whomever needs it can access it easily.

At the top of the tree is a one-page `READ-THIS-FIRST.md` (text file using CR-LF encoding, in case they view it on MS Windows). It's also printed as paper copies in the safes. This one-page document tells them about the "FOR HEIRS" USB flash drive, the safe deposit box, GnuCash, and various insurances.

Also in the safes is a one-page printout/PDF from GnuCash, which shows all my assets and liabilities, other than durable goods in the household. (I'm guessing those two one-pagers are most of what heirs would need, at least initially. If they need more later, there's the full GnuCash database, and the neatly named and organized PDF files of things like financial service statements.)

I've recently kludged an easy way to maintain household durable goods inventory, by kludging onto purchase transactions in GnuCash. I still have to find the time to encode all the cruft I've acquired in the past, though. If/once I do, a household inventory might become a third paper copy in the safes (for possible someday insurance or estate purposes).

This is a fascinating topic that doesn't get the attention it deserves.

The idea of trusting a single entity (the family computer expert, a single company, or a government agency) to be the sole host or custodian of important records and family photos/videos is fraught with risk: Even if that entity has redundancy across multiple geographic locations, they can have security breaches, their software can fail, their business can fail, or they can otherwise become incapacitated.

Security breach risk can be drastically reduced by encrypting your files before you even store them in that entity's system(s). But you'll need to protect that key and keep it in multiple locations that multiple people have access to. And it's going to make certain functions (like search indexing or file format conversions, to battle file format obsolescence) a lot more complicated.

Data loss risk due to failing hardware, software, companies, or people, can be greatly reduced by copying your stuff to multiple storage providers who ultimately rely on physically different data centers (e.g. if company A and B both ultimately use S3, Amazon is now my single point of failure), and taking an "append-only" approach.

But who is going to spend the time to pre-encrypt and sync all of your family's records like that? Possibly the family computer expert, who you can't depend on long-term.

If you're lucky, you have one who will do all of that for now, and maybe give everybody in the family a copy on a durable USB drive or similar* every few years. Or maybe you can pay someone to do the same. I'd pay for it, because it's a lot of work I don't want to do.

* For longer-term archival storage that should last several decades without power, M-Disc (DVDs or Blu-ray discs) used to be a go-to option. But that depends on people having access to a Blu-ray player decades from now, which I suspect will be a lot like the process of getting access to a floppy drive today.

My approach is to look at contents as data-first with tools on the top. A few simple patterns of file systems manage this. Right now, I use Dropbox[1] as a primary container, with a family’s Google Workspace[2] account to augment when it needs spreadsheets, documents, forms, etc. I use Insync[3] to convert and back up a copy in open formats for all documents from Google Workspace.

I have been doing this for quite a while for our family, but I have extended slowly to immediate relatives while we helped them out with technology-related tasks. I haven’t reached the stage of complexities with the files and folder hierarchy that might break cloud providers[4].

Our family has a complete offline copy of everything, including backups of the photos, videos, and others. Everything is still in the works, and I recently started digitizing some old pictures from the in-laws. I stumbled upon some of their photos from around India’s independence and played with the colorization and animation thingy (that popped up a few months back). It became a craze, and their super large family gathered for a presentation, making many elders cry with tears of joy and disbelief.

My most significant achievements, compliments, and moments of pride have been that almost all of the new generations in the family choose to study computer engineering or computer-related. And so far, the best one is the niece commenting, “When I grow up, I want to marry a boy who is as technical as aunty’s husband.”

I have copies of the commercial ones (companies set up, wound down) since their inception. I also try to stick to plain-text[5], and as open formats as possible.

1. https://www.dropbox.com/

2. https://workspace.google.com

3. https://www.insynchq.com

4. https://news.ycombinator.com/item?id=37507148

5. https://brajeshwar.com/2022/plain-text/

I wrote about this recently, mostly through the lens of the sentimentally important things like videos and photos https://joshvince.site/blog/20230903_little_metal_boxes

I have a folder where I store everything. All documents are named starting with the ISO date, and I make sure the filename contains the relevant keywords. All paper documents are scanned.

That’s it.

Many years ago I read of an organizational practice that I refer to as a "trash buffer".

I have a fairly deep paper bin that I lay papers flat in. Any marginal paper documents or receipts that aren't obviously garbage and aren't obviously valuable go into this bin, always placed on the top.

Every 10-12 months I pick up the stack and remove, then shred, the bottom 1/4 of it - without review or remorse. If I didn't need them by then I never would.

I find that I actually use this buffer:

Every month or two there actually is some marginal paper document that I need to shuffle through and find. It's in reverse chronological order so that is very simple.

More and more, I'm finding myself using Notion to track "life data stuff". I realize this is a very imperfect choice, but it offers the best tooling I've found so far for this sort of thing. I may migrate to some self-hosted open-source Notion clone at some point (there are a few).

I use paperless-ngx with 3-2-1 backup. It runs OCR and indexes them to makes everything searchable. I can go back years later and search for “toyota” find my car warranty agreement in an instance.

By points

1. Beancount via org-mode notes, org-babel for transactions, bean queries from the org mode file for reporting, non perfect, a new thing, in the past I've used GNU Cash but I prefer to put anything in org so...

2-5. equally in org-mode org roam managed notes, non textual stuff as org-attachments linked directly in notes or indirectly "elisp:(start-process-shell-command" alike to run different stuff than mere opening.

I have to retain something for fiscal reasons, something for personal reasons and something are extras. It's not that perfectly crafted, but accessing anything by titles (org-roam-node-find) is like having anything in a graph, accessed via comfy search&narrow. When titles fails consult-rg on the org-mode root (~/notes) does the rest.

I've tries org-ql, to query notes for certain infos like "active contracts I have", but even with templating (yasnippet) it's not that immediate, I have some queries and templates but I do not really use if not very rarely. I've try LLama (Khoj) on my notes but well, results are very bad compared to direct textual mach access. So, yes, I have some clutter, I do not see it much, it does not consume significant amount of storage nor pollute my searches so... It's manageable, rarely I found something is the clutter, so it's not totally useless anyway. I am able to find anything noted so far, of course I do not log my life completely so sometimes I'm looking for things I do not have, that's rare but happen. In this cases the answer is "it depend".

I choose to live digitally in Emacs mostly because of integration: I have mails, todos, notes, anything in the same place/tool/model/paradigm like my mind is one for all.

Downsides: it's a personal thing, not really usable by other family members, not designed for collaborative works, it's possible works together but only as ISOLATED individual, all on their own desktop, possibly Emacs but not necessarily and definitively not "the same Emacs" sharing steps/results by various means like patches via mail, direct textual mails "I found this and that, I conclude that" etc.

The biggest “aha!” thing for me has not been any complicated data lake, but rather one simple text file.

In this file I write the date and a short note anytime something important happens. Visit the doctor and get test results? Write a note. Change the oil in my car? Write a note. Get a quote for a mortgage? Write a note. Open a new bank account? Write a note. Shop for new running shoes and learn what size fits best? Write a note. Have a contractor come work on the house? Write a note.

If there are important files that relate to those I list filenames and say where they are, but frankly I haven’t found that to be very useful. Rarely do I need any original documents. And if I died prematurely, my family can just scroll through this document to check off accounts (Google will automatically send it to them if I don’t access it for a period of time).

I’ve been using it for a few years and found it to be a total lifesaver. I frequently just need to quickly see how I did something, who I used, or his much something cost.

I store the live version on Google Drive so I can edit it anywhere, but you could host it locally.

I'm very grateful to everyone for sharing their thoughts and suggestions on a topic that is near-and-dear to my heart.

Does anyone have access to an AI into which they could copy-paste this whole page and have it extract a ranked list of the products that people mentioned, perhaps along with links and counts of people who mentioned them?

The page size exceeds the limit of what free gpt3.5 will do - so, just asking before trying to do it chunk by chunk.