Thiago Baldim Thiago Baldim - 28 days ago 21
Python Question

Connecting Kafka-Python with a cluster with Kerberos

I'm trying to connect to a Kafka with Kafka-Python, the Kafka cluster has Kerberos that we need to build some commands to do few steps.

I have created one Topic at the cluster and I did the test with

./kafka-console-producer.sh
and
./kafka-console-consumer.sh
and works really well.

But when I try to connect with Kafka-Python I had a problem. See my code below:

def produce():
print ('Producer')
k_producer = KafkaProducer(bootstrap_servers='hostname:6667',
security_protocol='SASL_PLAINTEXT',
sasl_plain_username='machine_usr',
sasl_plain_password='machine_pwd',
sasl_mechanism='PLAIN')
for i in range(10):
print ('Before send')
k_producer.send('myTopic', 'Testing My Topic ' + str(i))
print ('After send')


Well, running this stuff I got this erro message after 30 secconds:

File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\producer\kafka.py", line 328, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 202, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 791, in check_version
kafka.errors.NoBrokersAvailable: NoBrokersAvailable


I'm running it in a remote machine. And the bootstrap_server I used the Zookeeper hostname and port but didn't work as well.

I found few things about it, and in the git of the Kafka-Python I found that they had implemented.

Is there other way to connect?

Answer

Well Guys,

I found the issue.

The problem is that Kerberos is not supported for Kafka producer in Python using Key Tab.

To use Key Tab we need to set a java Environment Variable.

According Hortonworks we need to set the client_jaas_client to connect.

The solution was using Py4j to call the Kafka Producer in JVM.

For new Python will not work.