Zefu Hu Zefu Hu - 1 month ago 11
Scala Question

Can Apache Kafka send non-string messages through a topic?

A mllib model is trained somewhere and I want it to be sent to somewhere else. When I try to send it through a kafka topic like this

val model = LogisticRegressionModel.load(sc, "/PATH/To/Model")
val producer=new Producer[String, LogisticRegressionModel](config)
val data=new KeyedMessage[String, LogisticRegressionModel(topic2,key,model)
producer.send(data)
producer.close()


I would encounter an error like this:

org.apache.spark.mllib.classification.LogisticRegressionModel cannot be cast to java.lang.String


So, is it possible for kafka to send non-string messages through a topic?

Answer

You can send non-string messages to Kafka topic using Kafka Producer. From 0.9.0 version its better to use Java Client instead of Scala Client.

All you need to do is specifying the correct Key, Value serializer in Properties like below.

props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Comments