python connect kafka

       Thought:

       Write to the kafka cluster,

       read to zk

       Be sure to start looking for python to connect to the standard library of kafka. The former kafka-python and pykafka use more mature libraries, and the latter is an upgraded version of Samsa. Go to the article on the Internet to connect in python and use kafka to use samsa to connect to zookeeper and then Using kafka Cluster can meet my needs very well. In the example of pykafka, I also see the support of zk, but kafka-python does not have support for zk, so I chose pykafka as the connection library  

conceptual problem

For the cluster of kafaka and zookeeper, when samsa is used, both the producer and the consumer are connected to zookeeper, but I communicate with Fengyun (big data giant, operation and maintenance diaosi reversal), when they use it, the producer directly connects to the kafaka server list , consumers only use zookeeper. This also solves the confusion that I read the pykafka documentation, only consumers can connect to zookeeper, so the problem is solved, just follow the documentation.

producer

1 2 3 4 5 6

>> > from pykafka import KafkaClient >> > client = KafkaClient (hosts= "192.168.1.1:9092, 192.168.1.2:9092" ) # Multiple clients are acceptable. This is the point >> > client.topics # View all topics > > > topic = client.topics[ 'my.test' ] # select a topic >> > producer = topic.get_producer() >> > producer.produce([ 'test message ' + str(i ** 2 ) for i in range( 4 )]) # Added a str official example py2.7 can't run

consumer

1 2 3 4 5

>> > balanced_consumer = topic.get_balanced_consumer( consumer_group= 'testgroup' , auto_commit_enable= True , # No need to add consumer_group when set to Flase zookeeper_connect= 'myZkClusterNode1.com:2181,myZkClusterNode2.com:2181/myZkChroot' # Here is the connection multiple zk )