smeeb smeeb - 10 months ago 70
Java Question

When does Kafka connection require ZooKeeper config?

The Kafka console consumer seems to require you to specify a ZooKeeper instance to connect to:

./ --zookeeper --topic mytopic

But it is clearly possible to connect to a Kafka broker directly via the Java API:

public class KafkaClient {
public static void main(String[] args) {

String topic = "mytopic";

Properties props = new Properties();
props.put("bootstrap.servers", "");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Producer<String, String> producer = new KafkaProducer<>(props);

Callback cb = new Callback() {
void onCompletion(RecordMetadata rdata, Exception exc) {
if(exc) {
throw exc;

producer.send(new ProducerRecord<String, String>(topic, 'somekey', 'someval'), cb);

Is there a way to run the consumer without specifying a ZK node? If not, why?

Answer Source

This depends on which version of the consumer API is being used. Starting with the latest Kafka release 0.10.1, the new API which directly targets the brokers is the default used by the console consumer. Versions prior to 0.10.1 default to the older API targeting Zookeeper but can be set to use the new consumer API for the console consumer by specifying parameters like: --new-consumer and --bootstrap-server someBroker:9092 with the command.