D. Krauchanka D. Krauchanka - 9 months ago 59
Java Question

How kafka balances partitions load?

i faced a question with load balancing in kafka. So, i created a topic with 10 partitions and created 2 consumers. The 10 partitions were divided and assigned to these consumers (5 partitions to the first and 5 to the second) and it works fine. Sometimes first consumer works, sometimes second.

But at one moment we can face a situation when for example second consumer receives a message and it takes time (for example 10 minutes) to handle this message.

So, my question is how kafka will decide to which partition store the message?

Round robin in this case i think is not a good idea, because messages in partitions that are handled by second consumer won't be handled until the second consumer finishes the long work.


According to the @Milan Baran answer, the load is balanced on the producer side. But in this case, even if we provide a custom

realization, it will be the same problem that the message that was stored in the partition which was assigned to the consumer that is doing long-term work, will not be processed until this consumer finishes its long-term work.

May be, there are additional load balancer somewhere else?

Answer Source

Thanks everybody for help. But i found an answer for my question. So first of all, there are at least 3 places where kafka balances load:

  1. To assign partitions to consumers "Round robin" or "Range" algorithms are used. This can be configured by setting the partition.assignment.strategy property. By default range is used.
  2. At producer level can be applied a strategy for selecting partition to store message. It can be done by partitioner.class
  3. And the answer for my question. If one consumer processes message for a long time, kafka thinks that this consumer is dead and reassign partitions between another consumers. So when a long time job is done by the consumer, no partitions are assigned to it. When the consumer finishes long time work partitions will be assigned to it again. And no messages will be pending.