Kafka单机部署

其他 2018-08-11 05:17:26 阅读次数: 0

一般而言，kafka集群和zookeeper集群是分开（分机）部署的，我们尝试一下。

所有配置文件下载地址：https://download.csdn.net/download/yongyu_it/10586668

1、参照《Zookeeper集群环境搭建》搭建zookeeper集群

环境配置如

$ pwd
/home/yong/Desktop/kafka-test-20180806/
$ tree .
.
├── cmd
│   ├── start1.sh
│   ├── start2.sh
│   └── start3.sh
└── docker-compose-zookeeper.yml

1 directory, 4 files

docker-compose-zookeeper.yml

version: '2'
 
networks:
  thinking:
  
services:
  test_1.thinking.com:
    image: ubuntu
    container_name: test_1.thinking.com
    command: sh /cmd/start1.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12
      - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin
 
 
  test_2.thinking.com:
    image: ubuntu
    container_name: test_2.thinking.com
    command: sh /cmd/start2.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12
      - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin
 
  test_3.thinking.com:
    image: ubuntu
    container_name: test_3.thinking.com
    command: sh /cmd/start3.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12
      - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin

start1.sh

mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 1 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bash

start2.sh

mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 2 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bash

start3.sh

mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 3 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bash

启动集群

$ docker-compose -f docker-compose-zookeeper.yml up -d
测试集群

$ docker exec -it test_1.thinking.com /bin/bash
root@ed717de94636:/# cd /home/zookeeper-3.4.12/bin
root@ed717de94636:/home/zookeeper-3.4.12/bin# sh zkCli.sh
Connecting to localhost:2181
2018-08-06 06:52:37,981 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-08-06 06:52:37,984 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=ed717de94636
2018-08-06 06:52:37,984 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_171
2018-08-06 06:52:37,986 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-08-06 06:52:37,986 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/zookeeper-3.4.12/bin/../build/classes:/home/zookeeper-3.4.12/bin/../build/lib/*.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/home/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/home/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/home/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/home/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/home/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/home/zookeeper-3.4.12/bin/../conf:
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.13.0-45-generic
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2018-08-06 06:52:37,988 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2018-08-06 06:52:37,988 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/zookeeper-3.4.12/bin
2018-08-06 06:52:37,989 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@255316f2
Welcome to ZooKeeper!
2018-08-06 06:52:38,005 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-08-06 06:52:38,068 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2018-08-06 06:52:38,104 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100003216670000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
create /zk-book 123
Created /zk-book
[zk: localhost:2181(CONNECTED) 1] get /zk-book
123
cZxid = 0x100000002
ctime = Mon Aug 06 06:52:44 GMT 2018
mZxid = 0x100000002
mtime = Mon Aug 06 06:52:44 GMT 2018
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 2] close
2018-08-06 06:52:55,164 [myid:] - INFO  [main:ZooKeeper@687] - Session: 0x100003216670000 closed
[zk: localhost:2181(CLOSED) 3] 2018-08-06 06:52:55,165 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@521] - EventThread shut down for session: 0x100003216670000
root@ed717de94636:/home/zookeeper-3.4.12/bin# 
$ docker exec -it test_2.thinking.com /bin/bash
root@53c259066ce4:/# cd /home/zookeeper-3.4.12/bin
root@53c259066ce4:/home/zookeeper-3.4.12/bin# sh zkCli.sh
Connecting to localhost:2181
2018-08-06 06:55:25,814 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-08-06 06:55:25,817 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=53c259066ce4
2018-08-06 06:55:25,817 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_171
2018-08-06 06:55:25,819 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/zookeeper-3.4.12/bin/../build/classes:/home/zookeeper-3.4.12/bin/../build/lib/*.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/home/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/home/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/home/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/home/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/home/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/home/zookeeper-3.4.12/bin/../conf:
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.13.0-45-generic
2018-08-06 06:55:25,821 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2018-08-06 06:55:25,821 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2018-08-06 06:55:25,821 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/zookeeper-3.4.12/bin
2018-08-06 06:55:25,822 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@255316f2
Welcome to ZooKeeper!
2018-08-06 06:55:25,843 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-08-06 06:55:25,913 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-08-06 06:55:25,939 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x200003212ac0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] get /zk-book
123
cZxid = 0x100000002
ctime = Mon Aug 06 06:52:44 GMT 2018
mZxid = 0x100000002
mtime = Mon Aug 06 06:52:44 GMT 2018
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 1] close
2018-08-06 06:55:36,662 [myid:] - INFO  [main:ZooKeeper@687] - Session: 0x200003212ac0000 closed
[zk: localhost:2181(CLOSED) 2] 2018-08-06 06:55:36,663 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@521] - EventThread shut down for session: 0x200003212ac0000
root@53c259066ce4:/home/zookeeper-3.4.12/bin#

2、启动kafka服务器（单机模式）

参考《kafka入门与实践》（牟大恩）

先编写docker compose启动脚本

$ pwd
/home/yong/Desktop/kafka-test-20180806/
$ tree .
.
├── docker-compose-kafka-sig.yml
└── kafka-sig-cmd
    └── kafka-sig.sh

1 directory, 2 files

docker-compose-kafka-sig.yml

version: '2'
 
networks:
  thinking:

services:
  kafka.thinking.com:
    image: ubuntu
    container_name: kafka.thinking.com
    command: sh /cmd/kafka-sig.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/kafka-test-20180806/kafka-sig-cmd:/cmd
      - /home/yong/Desktop/kafka_2.11-1.1.1:/usr/local/kafka
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin

改写kafka配置文件/usr/local/kafka/config/server. properties文件，主要是改写

broker.id

host.name

log.dirs

zookeeper.connect

broker.id=1
host.name=kafka.thinking.com
log.dirs=/tmp/kafka-logs
zookeeper.connect=test_1.thinking.com:2181,test_2.thinking.com:2181,test_3.thinking.com:2181

编写kafka-sig.sh文件

mkdir -p /tmp/kafka-logs
cd /usr/local/kafka/bin
ln -sf bash /bin/sh
./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
/bin/bash

启动kafka

$ docker-compose -f docker-compose-kafka-sig.yml up -d

检验kafka

$ docker exec -it kafka.thinking.com /bin/bash
# cd /usr/local/kafka/bin
# ./kafka-topics.sh --create --zookeeper test_1.thinking.com:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
# ./kafka-console-producer.sh --broker-list kafka.thinking.com:9092 --topic test
>Fuck you 1001
>Fuck you 1002
# ./kafka-console-consumer.sh --zookeepertest_1.thinking.com:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Fuck you 1001
Fuck you 1002
^CProcessed a total of 2 messages

成功！

猜你喜欢

转载自blog.csdn.net/YongYu_IT/article/details/81453880
今日推荐
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
周排行
浏览器对同一域名进行请求的最大并发连接数
React Hook之自定义Hook
【转】MyBatis缓存机制
-Java-泛型
自动化测试常用脚本-发送邮件
LeetCode#859: Buddy Strings
java、Python处理字符串
第二篇の博客
Hadoop伪分布式环境安装
SQL Server进阶（十一）临时表、表变量
每日归档
更多
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022