Any good reads, articles or books on file migration?

Question

I&rsquo;m looking to build a tool to migrate files between cloud providers (ie blob storage, s3, etc).I&rsquo;m looking to learn best practices for doing so. The volume is a few petabytes but can be done incrementally.I was thinking some queue that handles N transfers concurrently running in a k8s cluster. Queue I was thinking Redis or Kafka. However, I&rsquo;d love to build something nice and resilient

warrenm · Accepted Answer

Don't have any articles handy (beyond the various cloud providers' offerings which you can find with a quick "s3 to s3 migration" search)But the egress and ingress fees, added-onto the middleman position your tool would be occupying sound heavy :)

mattpallissard · Answer

This tool already exists and is fairly standard. No extra infra required.https://rclone.org/

Any good reads, articles or books on file migration?

Don't have any articles handy (beyond the various cloud providers' offerings which you can find with a quick "s3 to s3 migration" search)
But the egress and ingress fees, added-onto the middleman position your tool would be occupying sound heavy :)

This tool already exists and is fairly standard. No extra infra required.
https://rclone.org/