annedroiid annedroiid - 12 days ago 9
Java Question

KafkaConsumer never exits .poll method - GroupCoordinatorNotAvailableException

I have an implementation of a

KafkaConsumer
in java, and currently it is never exiting the
.poll
method. When I drill down into the source code in debug mode I've found that it is getting stuck in the while loop in
AbstractCoordinator.ensureCoordinatorKnown()
, as the coordinator is never found.

The future returned from
sendGroupMetadataRequest()
in the loop fails the first time with
org.apache.kafka.clients.consumer.internals.SendFailedException
, and then will fail every subsequent time with
org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The group coordinator is not available.
. Does anyone know why this might happen?

If I use the console producer/consumer I am able to successfully send and receive messages, it is only when I use my implementation of the KafkaConsumer. Additionally, the consumer does work on two of my servers so I know it is not the implementation of the consumer.

Here are the properties my consumer is created with:

Properties props = new Properties();
props.put("bootstrap.servers", "myserver:9000);
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("group.id", groupId);
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");


Edit:

The topic is definitely created before the consumer starts.

Edit 2:
I deleted all of the brokers in my cluster and recreated them, and now I'm failing at a different point. In
AbstractCoordinator.ensureActiveGroup()
while trying to rejoin, the future returned from
performGroupJoin()
repeatedly fails with
org.apache.kafka.common.errors.NotCoordinatorForGroupException: This is not the correct coordinator for this group.
. Still not sure what is going on.

Edit 3:
I deleted the brokers and recreated them with a different id and now the
.poll()
method is returning and it's successfully consuming messages. I'd still like to know why it failed in the first place though so I can make sure it doesn't happen again.

Answer

Deleting the brokers and creating new ones fixed the problem. Still not sure went wrong with the brokers though.

Comments