Good pub/sub setup for personal projects?

Question

I&rsquo;m messing around with a single node k8s homelab I&rsquo;m running for fun side projects and micro service development.I&rsquo;ve done a lot of work with Kafka, but it&rsquo;s pretty heavy and I really just want a decent pub/sub queue setup between services that would be somewhat reliable.Is there anything you&rsquo;d recommend? Open to thoughts on just using Postgres as a queue as well.

boloust · Accepted Answer

At one end of the spectrum: ZeroMQ[1]. No broker to run, and everything can stay within your local network. Just add the library to your microservices and expose a port. It gives you a socket interface with well-engineered queuing primitives. It's enough for most side projects in its simplest form, while also giving you the flexibility to implement a huge number of distributed queuing patterns.
At the other end of the spectrum: just use a managed queue from your preferred big cloud provider. All are reliable enough for this use case, and the cost for most projects would be zero or negligible.
[1]: https://zeromq.org/

rektide · Answer

Not entirely a real suggestion, but if you want really ultra-fun & easy, patchbay is the most awesome, simplest, easiest-to-use & most versatile/multi-purpose http service for relaying things around/queuing that I know.
Atm it's all in-memory, afaik. There are go & node.js implementations, which I hope some-day we add persistence to.
I'm still working on k8s Persistent Volumes, but in terms of bigger/better software, I hope soon-ish to have a Pulsar Operator[2] going. Kafka is very very exacting & specific, inflexible, but Pulsar's architecture seems much more general & flexible, capable of things like great geo-distribution by virtue of it's underlying Bookkeeper storage layer. It'd be my production go-to for any pubsub details.
[1] https://github.com/patchbay-pub https://patchbay.pub/ https://news.ycombinator.com/item?id=21639066 (362 points, 3 years ago, 79 comments)
[2] https://docs.streamnative.io/operators/pulsar-operator/pulsa...

remram · Answer

Redis is extremely easy (single binary), though that might not be super compelling if you can run containers. RabbitMQ is good too and has additional features like automatic requeueing if the client stops before acknowledging (and more complex routing rules etc). Both have clients in many languages.

fendy3002 · Answer

uhm, rabbitMQ?the problem is just Kafka being heavy?

pramsey · Answer

I had a good time with the pq module for a pgsql/python [1] queue. There's a certain brutal simplicity to just having one persistent process, the database, to manage.[1] https://pypi.org/project/pq/

winrid · Answer

Why do you want to use a queue between services for a personal project? Just use RPC or similar?Unless you want to play with queues... otherwise I would use the database, and add redis or rabbitmq later.

wdb · Answer

Maybe Redpanda or Redis pub/sub?

joshxyz · Answer

redis streams are cool

Good pub/sub setup for personal projects?

Redis is extremely easy (single binary), though that might not be super compelling if you can run containers. RabbitMQ is good too and has additional features like automatic requeueing if the client stops before acknowledging (and more complex routing rules etc). Both have clients in many languages.

uhm, rabbitMQ?
the problem is just Kafka being heavy?

I had a good time with the pq module for a pgsql/python [1] queue. There's a certain brutal simplicity to just having one persistent process, the database, to manage.
[1] https://pypi.org/project/pq/

Why do you want to use a queue between services for a personal project? Just use RPC or similar?
Unless you want to play with queues... otherwise I would use the database, and add redis or rabbitmq later.

Maybe Redpanda or Redis pub/sub?

redis streams are cool