i faced a question with load balancing in kafka. So, i created a topic with 10 partitions and created 2 consumers. The 10 partitions were divided and assigned to these consumers (5 partitions to the first and 5 to the second) and it works fine. Sometimes first consumer works, sometimes second.
But at one moment we can face a situation when for example second consumer receives a message and it takes time (for example 10 minutes) to handle this message.
So, my question is how kafka will decide to which partition store the message?
Round robin in this case i think is not a good idea, because messages in partitions that are handled by second consumer won't be handled until the second consumer finishes the long work.
According to the @Milan Baran answer, the load is balanced on the producer side. But in this case, even if we provide a custom
Thanks everybody for help. But i found an answer for my question. So first of all, there are at least 3 places where kafka balances load:
partition.assignment.strategyproperty. By default range is used.