What do you use to move (copy) data between systems?

Question

Do you centralize your data? Do you run an iPaaS of some kind? Do you like it?Context: I'm currently in the middle of a project where I'm hooking up an ERP system (SalesPad) to an eCommerce website (WooCommerce) and I've found I need to move many different kinds of data between them (Inventory/Orders/CustomPricing/etc), and I haven't found a good tool so I started building one.Has anyone else run into this? How did you solve it? Did you like your solution?

eb0la · Accepted Answer

Take a look at kafka ;-) Just put the file path as the key and the content as the message value.You get TTL/size garbage collection for free: no need to get/develop a sophisticated client/server.

sitkack · Answer

rsync, rclonereally depends on what formats the data is in, how it will be used, how long it needs to last, how long the code that reads and writes the data will last, how much out of band information vs inband information will be sent along with the data.You talked about movement, movement is the last of your worries, formats and provenance should rank higher on your list of concerns.My gotos are sqlite, xml and avro. Try to keep things under 2GB so you don't run into lame data transfer issues.

linsomniac · Answer

Many of these sorts of things we will push to an S3 bucket, and have other parts of the system pull from that bucket. We also use "rsync" with SSH keys to copy some data. Some of our vendors provide data to us via FTP, similarly it could be provided via HTTPMaybe Zapier can help?

Jugurtha · Answer

Can you configure database replication so your database handles the transfer to the target system, and then make the ecommerce site accessible for orders? The two databases will be kept in sync.

toomuchtodo · Answer

Have you checked out http://n8n.io? Sounds like you&rsquo;re just pushing JSON blobs around on a bus.

beeve · Answer

Lots of data integration software is available. In the open source world, Apache Nifi and Apache Camel are two examples.

alexchantavy · Answer

To add to the rsync answers, I think robocopy in a Windows environment is great.

nailer · Answer

rsync. Because it's efficient and well understood and emphasises accuracy.

What do you use to move (copy) data between systems?

Take a look at kafka ;-) Just put the file path as the key and the content as the message value.
You get TTL/size garbage collection for free: no need to get/develop a sophisticated client/server.

Many of these sorts of things we will push to an S3 bucket, and have other parts of the system pull from that bucket. We also use "rsync" with SSH keys to copy some data. Some of our vendors provide data to us via FTP, similarly it could be provided via HTTP
Maybe Zapier can help?

Can you configure database replication so your database handles the transfer to the target system, and then make the ecommerce site accessible for orders? The two databases will be kept in sync.

Have you checked out http://n8n.io? Sounds like you’re just pushing JSON blobs around on a bus.

Lots of data integration software is available. In the open source world, Apache Nifi and Apache Camel are two examples.

To add to the rsync answers, I think robocopy in a Windows environment is great.

rsync. Because it's efficient and well understood and emphasises accuracy.