HACKER Q&A
📣 2upmedia

WebSocket data architecture strategies: idempodent vs. incremental data?


How have you guys architected your data over websockets? Do you use certain strategies to reduce latency with less data over the wire and/or strategies to ensure local memory doesn't exceed a certain amount?

  👤 bem Accepted Answer ✓
I think you could find some inspiration in cryptocurrency exchanges. Most of them expose public websockets for prices, order books, trades, etc. They're high volume with lots of subscribers.

For example, check out the docs for the Binance websockets (they use both snapshot and delta messages): https://github.com/binance/binance-spot-api-docs/blob/master...

If you could tell me a little more about the data format, data volumes, number of subscribers, and how you get the data on the backend, I can try to give you some more concrete advice.

👤 aranchelk
If I understand the question: you don’t have to choose between idempotence and efficient incremental updates if you can give your application data a revision number. Send your incremental update(delta) with a reference to the app data’s last revision. Increment the revision number after applying each delta and reject any delta that doesn’t reference the current version on the server, they’ll only be applied at most once, so they’re idempotent.

There are fancier versions of this scheme that can be devised based on the business rules for your application.

👤 colesantiago
How are you currently doing this?

did you try going for a 3rd party provider like Firebase, Ably or Pubnub?