M. Carlson M. Carlson - 9 months ago 105
Java Question

Kafka Connect implementation errors

I was running through the tutorial here: http://kafka.apache.org/documentation.html#introduction

When I get to "Step 7: Use Kafka Connect to import/export data" and attempt to start two connectors I am getting the following errors:

ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding messages, 1 left

ERROR Failed to commit offsets for WorkerSourceTask

Here is the portion of the tutorial:

Next, we'll start two connectors running in standalone mode, which means they run in a single, local, dedicated process. We provide three configuration files as parameters. The first is always the configuration for the Kafka Connect process, containing common configuration such as the Kafka brokers to connect to and the serialization format for data. The remaining configuration files each specify a connector to create. These files include a unique connector name, the connector class to instantiate, and any other configuration required by the connector.
bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties

I have spent some time looking for a solution, but was unable to find anything useful. Any help is appreciated.


Answer Source

The reason I was getting this error was because the first server I created using the config/server.properties was not running. I am assuming that because it is the lead of the topic, the messages could not be flushed and the offsets could not be committed.

Once I started the kafka server using the server propertes (config/server.properties) This issue was resolved.