Write data from the console and write it back to the console is a convenient starting point, but you might want to use data from other sources or from Kafka to export data to other systems. For many systems, you may be used Kafka Connect import or export data, without writing custom integration code.
Kafka Kafka Connect is included with tools for data import and export to Kafka. It is scalable running tool connector, the connector is implemented for performing custom logic to interact with external systems. In this quick start, we will see how to use a simple connector to run Kafka Connect, the connector to import data from a file to Kafka theme and export the data to a file from Kafka theme.
First, we will create some seed data begin testing:
echo -e "foo\nbar" > test.txt
Next, we will start with two separate modes running the connector , which means they will run in a single local private process. We offer three configuration file as a parameter. The first is always Kafka Connect configuration process, which comprises a general configuration, such as a sequence of data format and Kafka agents to be connected. The rest are designated connector configuration file to be created. These include a unique name of a connector, any other configuration desired to be instantiated class connectors, and a connector.
[root@zookeep-kafka-node1 kafka_2.12-2.3.0]# bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
As error may modify connect-standalone.properties file bootstrap.servers change the value of IP
# These are defaults. This file just demonstrates how to override some settings. bootstrap.servers=10.23.209.70:9092 # The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will # need to configure these based on the format they want their data in when loaded from or stored into Kafka
The sample configuration file (provided together with Kafka) before starting to use your default local cluster configuration, and create two connectors: a first source connector, which is read from the input file to generate a row and each row Kafka theme, the second connector is a sink. Reading messages from Kafka topic, and they are displayed as a row in the output file.
During startup, you will see a lot of log messages, including some news show is instantiated connector. Kafka Connect After the process starts, it should start from the source connector test.txtrelating to the read line and to generate a themeconnect-test , and the receiver connector should start reading the message from the topic connect-test and write files test.sink.txt. We can verify this by checking the contents of the output data file has been passed through the entire pipeline:
[root@zookeep-kafka-node1 kafka_2.12-2.3.0]# more test.sink.txt foo bar
Note that the data stored in Kafka topic connect-test, so we can run the console user to view the data in this topic (or use a custom user code processing):
[root@zookeep-kafka-node2 kafka_2.12-2.3.0]# bin/kafka-console-consumer.sh --bootstrap-server 10.23.209.71:9092 --topic connect-test --from-beginning {"schema":{"type":"string","optional":false},"payload":"foo"} {"schema":{"type":"string","optional":false},"payload":"bar"}
The connector continues processing the data, so we can add data to a file and view it in the movement of the pipeline:
[root@zookeep-kafka-node1 kafka_2.12-2.3.0]# echo "hello world" >>test.txt
You should see this line appears in the console output and the user receivers file.
[root@zookeep-kafka-node2 kafka_2.12-2.3.0]# bin/kafka-console-consumer.sh --bootstrap-server 10.23.209.71:9092 --topic connect-test --from-beginning {"schema":{"type":"string","optional":false},"payload":"foo"} {"schema":{"type":"string","optional":false},"payload":"bar"} {"schema":{"type":"string","optional":false},"payload":"hello world"}