What big problems with Apache Kafka do you have?

Question

Hey folks,Me, with a bunch of friends, have some expertise in kafka ecosystem and wanna do something cool & open-source.The thing is, we can't decide what to do. Like, let's say there are already some solutions for some problems, a lot of web/standalone UI clients out there. For example, I know managing connectors with k8s can be a problem and existing solutions (like confluent's) are not ideal, but not sure if that's really a big problem for most of people.Please share your thoughts. Do you have any problems related to kafka ecosystem really bugging you? What could be done better?

MaknMoreGtnLess · Accepted Answer

The JVM is a blackbox that fails in terrifying ways at the worst time possible.
You won't know this until you take your service to production and expose it to real life traffic.
Now days, unless there's an absolute non-alternative to a Java project, I stay away from one.
For example, for most cases, Apache Kafka can be replaced by RedPanda (C++). Other potential solutions are LiftBridge (or core NATS)

gertd · Answer

I find topic balancing and scaling up and down volumes related to that a royal challenge resulting in having to balance manually to remove/reduce resource bottlenecks.