kafka生产与消费

其他 2018-06-09 05:07:57 阅读次数: 2

编写测试代码

无论使用哪种语言操作kafka其本质上都是在围绕两个角色进行的，分别是Producer、Consumer

已经在kafka boker里面创建好一个Topic,


1、创建Produer

1）、命令行方式---普通的发送方式

[python]  view plain  copy
  1. [root@node1 python_app]# python   
  2. Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)   
  3. [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2  
  4. Type "help", "copyright", "credits" or "license" for more information.  
  5. >>> from kafka import KafkaProducer  
  6. >>> producer = KafkaProducer(bootstrap_servers='192.168.120.11:9092')  
  7. >>> for _ in range(100):  
  8. ...     producer.send('world',b'some_message_bytes')  
  9. ...   
  10. <kafka.producer.future.FutureRecordMetadata object at 0xddb5d0>  
  11. <kafka.producer.future.FutureRecordMetadata object at 0xddb750>  
  12. <kafka.producer.future.FutureRecordMetadata object at 0xddb790>  

上面的几行功能分别是：

导入KafkaProducer

创建连接到192.168.120.11:9092这个Broker的Producer,

循环向world这个Topic发送100个消息，消息内容都是some_message_bytes'，这种发送方式不指定Partition，kafka会均匀大把这些消息分别写入5个Partiton里面，


更详细的说明可以参考 https://kafka-python.readthedocs.io/en/master/index.html


2）、命令行方式---发送json字符串

json作为一种强大的文本格式，已经得到非常普遍的应用，kafak-python也支持发送json格式的消息

其实如果你参考https://kafka-python.readthedocs.io/en/master/index.html这里的KafkaProducer里面的发送json

一定会报错的，这应该是这个文档的一个bug，

[python]  view plain  copy
  1. >>> producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))  
  2. Traceback (most recent call last):  
  3.   File "<stdin>", line 1, in <module>  
  4.   File "/usr/lib/python2.6/site-packages/kafka_python-1.3.4-py2.6.egg/kafka/producer/kafka.py", line 347, in __init__  
  5.     **self.config)  
  6.   File "/usr/lib/python2.6/site-packages/kafka_python-1.3.4-py2.6.egg/kafka/client_async.py", line 220, in __init__  
  7.     self.config['api_version'] = self.check_version(timeout=check_timeout)  
  8.   File "/usr/lib/python2.6/site-packages/kafka_python-1.3.4-py2.6.egg/kafka/client_async.py", line 841, in check_version  
  9.     raise Errors.NoBrokersAvailable()  
  10. kafka.errors.NoBrokersAvailable: NoBrokersAvailable  
经过测试，应该是这样写才是OK的

[python]  view plain  copy
  1. >>> producer = KafkaProducer(bootstrap_servers='192.168.120.11:9092',value_serializer=lambda v: json.dumps(v).encode('utf-8'))  
  2. >>> producer.send('world', {'key1': 'value1'})  
  3. <kafka.producer.future.FutureRecordMetadata object at 0x2a9ebd0>  
  4. >>>   

3）、命令行方式---发送普通字符串

[python]  view plain  copy
  1. >>> producer.send('world', key=b'foo', value=b'bar')  
  2. <kafka.producer.future.FutureRecordMetadata object at 0x29dcd90>  
  3. >>>   

4）、命令行方式--发送压缩字符串

[python]  view plain  copy
  1. >>> producer = KafkaProducer(bootstrap_servers='192.168.120.11:9092',compression_type='gzip')  
  2. >>> producer.send('world', b'msg 1')  

经过测试这种方式发送的内容，在接收方收到的消息仍然是普通的字符串，也许是没有安装python-lz4，原文中有这样的内容：

kafka-python supports gzip compression/decompression natively. To produce or consume lz4 compressed messages, you should install python-lz4 (pip install lz4). To enable snappy, install python-snappy (also requires snappy library). See Installation for more information.


上面都是测试各个命令的使用，接下来，我们写一个完整的脚本，这个脚本的功能是把指定目录下的文件名发送到world这个topic

file_monitor.py脚本

[python]  view plain  copy
  1. #-*- coding: utf-8 -*-  
  2.   
  3. from kafka import KafkaProducer  
  4. import json  
  5. import os  
  6. import time  
  7. from sys import argv  
  8.   
  9. producer = KafkaProducer(bootstrap_servers='192.168.120.11:9092')  
  10.   
  11. def log(str):  
  12.     t = time.strftime(r"%Y-%m-%d_%H-%M-%S",time.localtime())  
  13.     print("[%s]%s"%(t,str))  
  14.   
  15. def list_file(path):  
  16.     dir_list = os.listdir(path);  
  17.     for f in dir_list:  
  18.          producer.send('world',f)  
  19.          producer.flush()  
  20.          log('send: %s' % (f))    
  21.   
  22. list_file(argv[1])  
  23. producer.close()  
  24. log('done')  


假如我们要监控/opt/jdk1.8.0_91/lib/missioncontrol/features这个目录下的文件，可以这样执行

python file_monitor.py  /opt/jdk1.8.0_91/lib/missioncontrol/features

执行结果如下：

[python]  view plain  copy
  1. [root@node2 python_app]# python file_monitor.py  /opt/jdk1.8.0_91/lib/missioncontrol/features  
  2. [2017-11-07_17-41-04]send: org.eclipse.ecf.filetransfer.ssl.feature_1.0.0.v20140827-1444  
  3. [2017-11-07_17-41-04]send: org.eclipse.emf.common_2.10.1.v20140901-1043  
  4. [2017-11-07_17-41-04]send: com.jrockit.mc.feature.rcp.ja_5.5.0.165303  
  5. [2017-11-07_17-41-04]send: com.jrockit.mc.feature.console_5.5.0.165303  
  6. [2017-11-07_17-41-04]send: org.eclipse.ecf.core.feature_1.1.0.v20140827-1444  
  7. [2017-11-07_17-41-04]send: org.eclipse.equinox.p2.core.feature_1.3.0.v20140523-0116  
  8. [2017-11-07_17-41-04]send: org.eclipse.ecf.filetransfer.httpclient4.ssl.feature_1.0.0.v20140827-1444  
  9. [2017-11-07_17-41-04]send: com.jrockit.mc.feature.rcp_5.5.0.165303  
  10. [2017-11-07_17-41-04]send: org.eclipse.babel.nls_eclipse_zh_4.4.0.v20140623020002  
  11. [2017-11-07_17-41-04]send: com.jrockit.mc.rcp.product_5.5.0.165303  
  12. [2017-11-07_17-41-04]send: org.eclipse.help_2.0.102.v20141007-2301  
  13. [2017-11-07_17-41-04]send: org.eclipse.ecf.core.ssl.feature_1.0.0.v20140827-1444  
  14. [2017-11-07_17-41-04]send: org.eclipse.ecf.filetransfer.httpclient4.feature_3.9.1.v20140827-1444  
  15. [2017-11-07_17-41-04]send: org.eclipse.e4.rcp_1.3.100.v20141007-2033  
  16. [2017-11-07_17-41-04]send: org.eclipse.babel.nls_eclipse_ja_4.4.0.v20140623020002  
  17. [2017-11-07_17-41-04]send: com.jrockit.mc.feature.flightrecorder_5.5.0.165303  
  18. [2017-11-07_17-41-04]send: org.eclipse.emf.ecore_2.10.1.v20140901-1043  
  19. [2017-11-07_17-41-04]send: org.eclipse.equinox.p2.rcp.feature_1.2.0.v20140523-0116  
  20. [2017-11-07_17-41-04]send: org.eclipse.ecf.filetransfer.feature_3.9.0.v20140827-1444  
  21. [2017-11-07_17-41-04]send: com.jrockit.mc.feature.core_5.5.0.165303  
  22. [2017-11-07_17-41-04]send: org.eclipse.rcp_4.4.0.v20141007-2301  
  23. [2017-11-07_17-41-04]send: com.jrockit.mc.feature.rcp.zh_CN_5.5.0.165303  
  24. [2017-11-07_17-41-04]done  


在consumer上看到的内容是这样：

[plain]  view plain  copy
  1. world:4:93: key=None value=org.eclipse.ecf.filetransfer.ssl.feature_1.0.0.v20140827-1444  
  2. world:1:112: key=None value=org.eclipse.emf.common_2.10.1.v20140901-1043  
  3. world:3:119: key=None value=com.jrockit.mc.feature.console_5.5.0.165303  
  4. world:1:113: key=None value=com.jrockit.mc.feature.rcp.ja_5.5.0.165303  
  5. world:0:86: key=None value=org.eclipse.ecf.core.feature_1.1.0.v20140827-1444  
  6. world:1:114: key=None value=org.eclipse.equinox.p2.core.feature_1.3.0.v20140523-0116  
  7. world:4:94: key=None value=org.eclipse.ecf.filetransfer.httpclient4.ssl.feature_1.0.0.v20140827-1444  
  8. world:0:87: key=None value=com.jrockit.mc.feature.rcp_5.5.0.165303  
  9. world:4:95: key=None value=org.eclipse.babel.nls_eclipse_zh_4.4.0.v20140623020002  
  10. world:2:66: key=None value=com.jrockit.mc.rcp.product_5.5.0.165303  
  11. world:4:96: key=None value=org.eclipse.ecf.core.ssl.feature_1.0.0.v20140827-1444  
  12. world:2:67: key=None value=org.eclipse.help_2.0.102.v20141007-2301  
  13. world:1:115: key=None value=org.eclipse.ecf.filetransfer.httpclient4.feature_3.9.1.v20140827-1444  
  14. world:4:97: key=None value=org.eclipse.e4.rcp_1.3.100.v20141007-2033  
  15. world:0:88: key=None value=org.eclipse.babel.nls_eclipse_ja_4.4.0.v20140623020002  
  16. world:4:98: key=None value=com.jrockit.mc.feature.flightrecorder_5.5.0.165303  
  17. world:3:120: key=None value=org.eclipse.emf.ecore_2.10.1.v20140901-1043  
  18. world:1:116: key=None value=org.eclipse.equinox.p2.rcp.feature_1.2.0.v20140523-0116  
  19. world:4:99: key=None value=org.eclipse.ecf.filetransfer.feature_3.9.0.v20140827-1444  
  20. world:2:68: key=None value=com.jrockit.mc.feature.core_5.5.0.165303  
  21. world:4:100: key=None value=com.jrockit.mc.feature.rcp.zh_CN_5.5.0.165303  
  22. world:2:69: key=None value=org.eclipse.rcp_4.4.0.v20141007-2301  


2、创建Consumer

通常使用Kafka时会创建不同的Topic,并且在Topic里面创建多个Partiton,因此作为Consumer，通常是连接到指定的Broker,指定的Topic来消费消息。

完整的python 脚本

consumer.py

[python]  view plain  copy
  1. #-*- coding: utf-8 -*-  
  2.   
  3. from kafka import KafkaConsumer  
  4. import time  
  5.   
  6. def log(str):  
  7.         t = time.strftime(r"%Y-%m-%d_%H-%M-%S",time.localtime())  
  8.         print("[%s]%s"%(t,str))  
  9.   
  10. log('start consumer')  
  11. #消费192.168.120.11:9092上的world 这个Topic,指定consumer group是consumer-20171017  
  12. consumer=KafkaConsumer('world',group_id='consumer-20171017',bootstrap_servers=['192.168.120.11:9092'])  
  13. for msg in consumer:  
  14.         recv = "%s:%d:%d: key=%s value=%s" %(msg.topic,msg.partition,msg.offset,msg.key,msg.value)  
  15.         log(recv)  


重新启动file_monitor.py脚本

[python]  view plain  copy
  1. [root@node2 python_app]# python file_monitor.py  /opt/jdk1.8.0_91/lib/missioncontrol/features  
  2. [2017-11-07_18-00-31]send: org.eclipse.ecf.filetransfer.ssl.feature_1.0.0.v20140827-1444  
  3. [2017-11-07_18-00-31]send: org.eclipse.emf.common_2.10.1.v20140901-1043  
  4. [2017-11-07_18-00-31]send: com.jrockit.mc.feature.rcp.ja_5.5.0.165303  
  5. [2017-11-07_18-00-31]send: com.jrockit.mc.feature.console_5.5.0.165303  
  6. [2017-11-07_18-00-31]send: org.eclipse.ecf.core.feature_1.1.0.v20140827-1444  
  7. [2017-11-07_18-00-31]send: org.eclipse.equinox.p2.core.feature_1.3.0.v20140523-0116  
  8. [2017-11-07_18-00-31]send: org.eclipse.ecf.filetransfer.httpclient4.ssl.feature_1.0.0.v20140827-1444  
  9. [2017-11-07_18-00-31]send: com.jrockit.mc.feature.rcp_5.5.0.165303  
  10. [2017-11-07_18-00-31]send: org.eclipse.babel.nls_eclipse_zh_4.4.0.v20140623020002  
  11. [2017-11-07_18-00-31]send: com.jrockit.mc.rcp.product_5.5.0.165303  
  12. [2017-11-07_18-00-31]send: org.eclipse.help_2.0.102.v20141007-2301  
  13. [2017-11-07_18-00-31]send: org.eclipse.ecf.core.ssl.feature_1.0.0.v20140827-1444  
  14. [2017-11-07_18-00-31]send: org.eclipse.ecf.filetransfer.httpclient4.feature_3.9.1.v20140827-1444  
  15. [2017-11-07_18-00-31]send: org.eclipse.e4.rcp_1.3.100.v20141007-2033  
  16. [2017-11-07_18-00-31]send: org.eclipse.babel.nls_eclipse_ja_4.4.0.v20140623020002  
  17. [2017-11-07_18-00-31]send: com.jrockit.mc.feature.flightrecorder_5.5.0.165303  
  18. [2017-11-07_18-00-31]send: org.eclipse.emf.ecore_2.10.1.v20140901-1043  
  19. [2017-11-07_18-00-31]send: org.eclipse.equinox.p2.rcp.feature_1.2.0.v20140523-0116  
  20. [2017-11-07_18-00-31]send: org.eclipse.ecf.filetransfer.feature_3.9.0.v20140827-1444  
  21. [2017-11-07_18-00-31]send: com.jrockit.mc.feature.core_5.5.0.165303  
  22. [2017-11-07_18-00-31]send: org.eclipse.rcp_4.4.0.v20141007-2301  
  23. [2017-11-07_18-00-31]send: com.jrockit.mc.feature.rcp.zh_CN_5.5.0.165303  
  24. [2017-11-07_18-00-31]done  

然后启动consumer.py脚本

[python]  view plain  copy
  1. [root@node1 python_app]# python consumer.py   
  2. [2017-09-23_11-34-00]start consumer  
  3. [2017-09-23_11-34-10]world:3:121: key=None value=org.eclipse.ecf.filetransfer.ssl.feature_1.0.0.v20140827-1444  
  4. [2017-09-23_11-34-10]world:2:70: key=None value=org.eclipse.emf.common_2.10.1.v20140901-1043  
  5. [2017-09-23_11-34-10]world:3:122: key=None value=com.jrockit.mc.feature.rcp.ja_5.5.0.165303  
  6. [2017-09-23_11-34-10]world:2:71: key=None value=com.jrockit.mc.feature.console_5.5.0.165303  
  7. [2017-09-23_11-34-10]world:0:89: key=None value=org.eclipse.ecf.core.feature_1.1.0.v20140827-1444  
  8. [2017-09-23_11-34-10]world:4:101: key=None value=org.eclipse.equinox.p2.core.feature_1.3.0.v20140523-0116  
  9. [2017-09-23_11-34-10]world:1:117: key=None value=org.eclipse.ecf.filetransfer.httpclient4.ssl.feature_1.0.0.v20140827-1444  
  10. [2017-09-23_11-34-10]world:2:72: key=None value=com.jrockit.mc.feature.rcp_5.5.0.165303  
  11. [2017-09-23_11-34-10]world:4:102: key=None value=org.eclipse.babel.nls_eclipse_zh_4.4.0.v20140623020002  
  12. [2017-09-23_11-34-10]world:2:73: key=None value=com.jrockit.mc.rcp.product_5.5.0.165303  
  13. [2017-09-23_11-34-10]world:3:123: key=None value=org.eclipse.help_2.0.102.v20141007-2301  
  14. [2017-09-23_11-34-10]world:3:124: key=None value=org.eclipse.ecf.core.ssl.feature_1.0.0.v20140827-1444  
  15. [2017-09-23_11-34-10]world:0:90: key=None value=com.jrockit.mc.feature.flightrecorder_5.5.0.165303  
  16. [2017-09-23_11-34-10]world:3:125: key=None value=org.eclipse.ecf.filetransfer.httpclient4.feature_3.9.1.v20140827-1444  
  17. [2017-09-23_11-34-10]world:3:126: key=None value=org.eclipse.e4.rcp_1.3.100.v20141007-2033  
  18. [2017-09-23_11-34-10]world:3:127: key=None value=org.eclipse.babel.nls_eclipse_ja_4.4.0.v20140623020002  
  19. [2017-09-23_11-34-10]world:2:74: key=None value=org.eclipse.emf.ecore_2.10.1.v20140901-1043  
  20. [2017-09-23_11-34-10]world:3:128: key=None value=org.eclipse.equinox.p2.rcp.feature_1.2.0.v20140523-0116  
  21. [2017-09-23_11-34-10]world:0:91: key=None value=com.jrockit.mc.feature.core_5.5.0.165303  
  22. [2017-09-23_11-34-10]world:3:129: key=None value=org.eclipse.ecf.filetransfer.feature_3.9.0.v20140827-1444  
  23. [2017-09-23_11-34-11]world:3:130: key=None value=org.eclipse.rcp_4.4.0.v20141007-2301  
  24. [2017-09-23_11-34-11]world:3:131: key=None value=com.jrockit.mc.feature.rcp.zh_CN_5.5.0.165303  

可以看到file_monitor.py脚本发送了一批文件名到word这个topic，并且consumer.py收到了这些文件名。

#####################################

转自：https://blog.csdn.net/see_you_see_me/article/details/78468421

猜你喜欢

转载自blog.csdn.net/qq_41262248/article/details/80207185
今日推荐
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
基本数据类型封装类比较 Java源码解读(一) 8种基本类型对应的封装类型
JS实现无缝滚动上
深入解析HashMap原理（基于JDK1.8）
mysql的连接池
关于.htc
linux下的ubuntu12.04图形界面
【数论】好推不好记的扩展欧几里德
设备树详解
cscope + tags 简单设置
xml学习
每日归档
更多
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022