annedroiid annedroiid - 1 year ago 219
Java Question

KafkaConsumer never exits .poll method - GroupCoordinatorNotAvailableException

I have an implementation of a

KafkaConsumer
in java, and currently it is never exiting the
.poll
method. When I drill down into the source code in debug mode I've found that it is getting stuck in the while loop in
AbstractCoordinator.ensureCoordinatorKnown()
, as the coordinator is never found.

The future returned from
sendGroupMetadataRequest()
in the loop fails the first time with
org.apache.kafka.clients.consumer.internals.SendFailedException
, and then will fail every subsequent time with
org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The group coordinator is not available.
. Does anyone know why this might happen?

If I use the console producer/consumer I am able to successfully send and receive messages, it is only when I use my implementation of the KafkaConsumer. Additionally, the consumer does work on two of my servers so I know it is not the implementation of the consumer.

Here are the properties my consumer is created with:

Properties props = new Properties();
props.put("bootstrap.servers", "myserver:9000);
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("group.id", groupId);
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");


Edit:

The topic is definitely created before the consumer starts.

Edit 2:
I deleted all of the brokers in my cluster and recreated them, and now I'm failing at a different point. In
AbstractCoordinator.ensureActiveGroup()
while trying to rejoin, the future returned from
performGroupJoin()
repeatedly fails with
org.apache.kafka.common.errors.NotCoordinatorForGroupException: This is not the correct coordinator for this group.
. Still not sure what is going on.

Edit 3:
I deleted the brokers and recreated them with a different id and now the
.poll()
method is returning and it's successfully consuming messages. I'd still like to know why it failed in the first place though so I can make sure it doesn't happen again.

Answer Source

Deleting the brokers and creating new ones fixed the problem. Still not sure went wrong with the brokers though.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download