HACKER Q&A
📣 rich_sasha

Which OSS message-passing framework?


TL;DR: which open-source framework is best suited for facilitating message passing between single client and single server over network, where reliably and correctness are top priority?

I have a client connecting to a server. Currently communication is via raw TCP / JSON messages. There are known reliability issues, and somewhat hacky suggestions of how to fix this.

Instead, I know open-source messaging frameworks exist, like ZeroMQ, RabbitMQ, Apache Kafka, probably others. They all advertise features like low-latency communication, routing, complex messaging topologies etc. These are not my primary concerns - I only have one server and one client. What I care about is

- Reliable delivery - messages are not silently dropped, and delivered in correct order

- Tolerance to crashes of either client or server (e.g. messages not consumed by the client get seamlessly replayed); e.g. the framework keeps track of messages written and read by client

- So long as latency, throughput, memory footprint etc. are not extremely unfavourable, these are non-objectives; typical required message throughput is less than 1 msg/sec, with maybe bursts of 10-20 of messages/sec for a few seconds.

- Ease of integration into a C++ (server) and Python (client) code base, and general ease of maintenance - if the framework is a very complex beast with a steep learning curve, it's not necessarily better than hand-crafting the TCP protocol.

Basically, I want a queue, into which server can push, client can pop from, and reliability is guaranteed by the framework. Does any OSS framework help my case? I tried googling around, but mostly get SEO garbage articles, or ones that focus on features I don't need. I also tried SE, but apparently asking about appropriate software choices is off-topic - go figure...


  👤 txutxu Accepted Answer ✓
At $work we use kafka (but always with replica 3).

But, after read your description, I could not recommend it to you, could be an operational burden...

The options that come to my mind (without entering into details): Redis pub/sub or PostgreSQL


👤 pestatije
All those frameworks should be good for your requirements...the fact they support more complex scenarios doesn't make them unsuitable for yours

👤 omani
you could use postgres with postgREST. push your JSONs via HTTP into a queue. a LISTEN/NOTIFY would trigger queue workers server-side.