I am trying to figure out how to properly write a simple producer application to Kafka.
1) Please tell me if I should avoid using 0.10.0.1, I am totally struggled with it.
This example I found from Apache wiki worked for 0.8
It’s completely different in 0.10
So this is what I am using:
"org.apache.kafka" % "kafka_2.10" % "0.10.0.1",
I am feeling very bad about this latest Kafka library, because apparently there are some known issues. In the runtime, it even shows a list of warnings about some configurations are not known config. But if I don’t provide them, the app won’t run. On github, people have identified this kind of issue.
2) Honestly, my code is super simple, if their documentation is even right.
val props = new Properties()
val producer = new KafkaProducer[String, String](props)
val t = System.currentTimeMillis()
val runtime = new Date().getTime();
val msg = runtime + ", hello world";
val message = new ProducerRecord[String, String](topic, null, msg)
Some backgrounds you'll find useful before using the producer: The old producer(Scala) in 0.8.x had already been removed starting 0.9.0.
Therefore, you are using the new producer(Java) now. Kafka dev team decides to remove any Zookeeper dependencies from the new client, no matter producer or consumer. So you should not set the ZK setting anymore. Besides, letting "advertised.listeners" point to ZK service is incorrect.
"advertised.listeners" is for IaaS machines which usually have more than one NICs. Here is what Kafka doc says about it:
In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for
listenerswill be used.
You could set this parameter to have client bind the public NIC, whereas the intra-communication for brokers binds the private NIC.