HACKER Q&A
📣 Chiff0

No-code data import pipeline builder, possibly a good idea?


I have an idea for a tool that would allow users to build data migration pipelines for their onboarding users from other (possibly competing) software. In my mind, it makes sense, since it would be great for UX, and, from my knowledge, data migrations are usually done manually and are just boring and complex. However, I haven’t really seen anyone doing that, at least successfully, so I wonder, why? My guess is that it’s really hard to to build a solution that could do something like that without much specific work on specific migrations, and the only way to find success is to basically be a consulting firm that does one-off integrations. Would love to hear your thoughts though.

Cheers!

  👤 PaulHoule Accepted Answer ✓
There is a certain viewpoint that ontology should be at the heart of "no code". If you think that you way you believe that the key part of applications programming is developing a mental model for the application domain and then constructing data structures, user interfaces and such around that model.

I've seen quite a few attempts at automatically generating mappings from one data model to another. In the age of tf-idf this was inevitably a disappointment. With something like sbert.net we can certainly do better.

I'd consider two scenarios: (i) migrating data from a production system to an analytics system, and (ii) migrating data from one production system to another.

Let's imagine the system is a hotel reservation system. In the case of (ii) any scrambling of the data could produce a very unhappy customer. You might test the migration multiple times but at some point you have to cut over the old system to the new system and live with the migration you did. In the case of (i) an error might throw off your analytics, but the analytics might help you find errors, and you can always feed back what you learn into the migration process and do it again. (In fact you'll probably 'migrate' again and again to make monthly, weekly reports.)