smeeb smeeb - 27 days ago 16
Java Question

When does Kafka connection require ZooKeeper config?

The Kafka console consumer seems to require you to specify a ZooKeeper instance to connect to:

./kafka-console-consumer.sh --zookeeper myzk.example.com:2181 --topic mytopic


But it is clearly possible to connect to a Kafka broker directly via the Java API:

public class KafkaClient {
public static void main(String[] args) {

String topic = "mytopic";

Properties props = new Properties();
props.put("bootstrap.servers", "kafka.example.com:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Producer<String, String> producer = new KafkaProducer<>(props);

Callback cb = new Callback() {
@Override
void onCompletion(RecordMetadata rdata, Exception exc) {
if(exc) {
throw exc;
}
}
}

producer.send(new ProducerRecord<String, String>(topic, 'somekey', 'someval'), cb);
producer.close();
}
}


Is there a way to run the consumer without specifying a ZK node? If not, why?

Answer

This depends on which version of the consumer API is being used. Starting with the latest Kafka release 0.10.1, the new API which directly targets the brokers is the default used by the console consumer. Versions prior to 0.10.1 default to the older API targeting Zookeeper but can be set to use the new consumer API for the console consumer by specifying parameters like: --new-consumer and --bootstrap-server someBroker:9092 with the command.