Best modern file transfer/synchronization protocol?

Question

Hi HN,I'm looking to build something to transfer files between two devices on a network (one client and one server, both of which are under my control).Obviously I could write something bespoke with raw TCP sockets, but rather than reinventing the wheel I'm curious about what existing options people recommend. I assume there are some better technologies than FTP nowadays?Ideally some kind of built-in fault tolerance would be great, because my plan is to use this on a phone/in an environment where the connection could be interrupted.Edit: just to clarify - this is something I want to build into an application I am writing, ideally with support across iOS (client), Windows, and mac (server).One way transfer is all I need, and I mostly plan on photos/videos (so multiple files ~3-20MB in size).Thanks!

orbz · Accepted Answer

No need to get fancy, scp or rsync are the tried and true options here.

ryukoposting · Answer

I use syncthing for this. It's a little fiddly to set up, and transfer speeds aren't great. However, it's very reliable once configured, and it barely uses any resources after an initial scan of the folder you want to sync.

2color · Answer

https://iroh.computer/sendmehttps://iroh.computer/Iroh is what you&rsquo;re looking for. You can embed into your code, and it handled wire encryption, verification, and NAT hole punching. All over QUIC.

pvtmert · Answer

I highly recommend Unison (https://github.com/bcpierce00/unison)
It allows you to sync between 2 machines (bi-directional) over TCP or SSH.
Note that TCP way is not encrypted, you may use wireguard as transport layer encryption for that purpose...
You can use an external application to copy if file size is larger than an arbitrary number. (Eg: use rsync for files > 1gb)

whalesalad · Answer

seconding rsync and syncthing.the server could expose an smb or nfs share, the client could mount it, and then sync to that mount.rsync over ssh also works, if you do not want to run smb/nfs.this is also a cool tool https://rclone.org/

rmorey · Answer

I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well

arun-mani-j · Answer

On a similar note, can someone tell me what's the fastest (wireless) way to transfer files between two laptops on same network (i.e. hotspot)?
scp, rsync, wormhole give me only 2-3 mb/s.
For the context, I'm trying to transfer about 50-70 GB files.
What's causing the bottleneck here and what am I missing? Thanks in advance!
https://github.com/magic-wormhole/magic-wormhole

oschrenk · Answer

If it&rsquo;s one way (that wasn&rsquo;t quite clear from the requirements to me).take a look at https://tus.io/

bhaney · Answer

Yeah I usually just use rsync for this. In a loop if the network is unreliable.

rrix2 · Answer

I built something on top of the Syncthing API this week after using it on its own for years.
A local instance of Syncthing can behave as a robust sync tool + inotify API for applications consuming the files: https://docs.syncthing.net/rest/events-get.html#get-rest-eve...
i believe there's an embeddable golang library, but if you want something easy to use on android check in on syncthing-fork which lets you define more granular sync conditions including "just turn on 5 minutes every hour" https://github.com/Catfriend1/syncthing-android

mynegation · Answer

What are your latency and bandwidth requirements? How big are the files? If you are already looking past obvious TCP-based choices like HTTP and FTP, you might be interested in FASP/Aspera https://en.m.wikipedia.org/wiki/Fast_and_Secure_ProtocolEdit: I&rsquo;ll leave it here just in case it is useful for others but it may or may not be embeddable into your app, especially on the phone.

jayknight · Answer

I would probably start with rsync.

sneak · Answer

rsync over ssh for one-shots.syncthing for continuous use.

smackeyacky · Answer

Grab an S3 bucket on amazon.Do a 3 way sync with the s3 command line tool.That way, you have a neat cloud backup as well. Wouldn't take any more than 20 minutes total to set up.

eternityforest · Answer

What about Jami? It runs almost everywhere and having an embeddable Jami library would be absolutely amazing, although a fair amount of work.

tripleo1 · Answer

1. warpinator, syncthing 2. rclone??--3. self hosted ipfs in tailscale or something? (that would be cool)

pluto_modadic · Answer

continuous sync - mutagen.io (maybe you could extract some of the libraries)depends on if it's large or small files.

Helmut10001 · Answer

rsync. If you are looking for a long term solution: zfs on both sides with zfs send.

yetanother12345 · Answer

eh... wget or curl ? or not modern enough for you?

toomim · Answer

Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...

Best modern file transfer/synchronization protocol?

No need to get fancy, scp or rsync are the tried and true options here.

I use syncthing for this. It's a little fiddly to set up, and transfer speeds aren't great. However, it's very reliable once configured, and it barely uses any resources after an initial scan of the folder you want to sync.

https://iroh.computer/sendme
https://iroh.computer/
Iroh is what you’re looking for. You can embed into your code, and it handled wire encryption, verification, and NAT hole punching. All over QUIC.

seconding rsync and syncthing.
the server could expose an smb or nfs share, the client could mount it, and then sync to that mount.
rsync over ssh also works, if you do not want to run smb/nfs.
this is also a cool tool https://rclone.org/

I cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well

If it’s one way (that wasn’t quite clear from the requirements to me).
take a look at https://tus.io/

Yeah I usually just use rsync for this. In a loop if the network is unreliable.

I would probably start with rsync.

rsync over ssh for one-shots.
syncthing for continuous use.

Grab an S3 bucket on amazon.
Do a 3 way sync with the s3 command line tool.
That way, you have a neat cloud backup as well. Wouldn't take any more than 20 minutes total to set up.

What about Jami? It runs almost everywhere and having an embeddable Jami library would be absolutely amazing, although a fair amount of work.

1. warpinator, syncthing 2. rclone??
--
3. self hosted ipfs in tailscale or something? (that would be cool)

continuous sync - mutagen.io (maybe you could extract some of the libraries)
depends on if it's large or small files.

rsync. If you are looking for a long term solution: zfs on both sides with zfs send.

eh... wget or curl ? or not modern enough for you?

Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...