I'm working on an IoT platform that has to receive, transform, store and visualize IoT data in a user-friendly dashboard. I'm currently using the Azure stack (stream analytics, data lake, SQL server, Power BI) but I'm running into major limitations and bugs so I'm looking for new tools that will help me out.
We're currently doing about 500M events per year, but this needs to scale WAY higher (the customer wants to retain detailed data for up to a year, but I question if that's realistic and if that's how others do it. Seems more sensible to me to aggregate data to compact it a bit). Whatever new setup I come up with has to be able to scale to at least 10B events per year, unless you all say it's impossible and I should just not try to store/visualize/analyze so much data. But I don't know what best-practices are.
I'm looking for ideas for what kind of tools I could use for storing the data, stream processing and for visualizing it all, and maybe some ideas for how they would work together.
Thanks a bunch in advance!
Without knowing what the domain is, what the events look like, etc: it's very hard to suggest solutions to the problem (because we don't know the problem)
Frankly it sounds like you need to partner with a software architect. Maybe Azure solutions engineers can help.