Storm ——集群搭建

服务器准备

192.168.214.150 centos6-1
192.168.214.151 centos6-2
192.168.214.152 centos6-3

安装启动Zookeeper

安装Storm

1.获取storm压缩包

[root@centos6-1 ~]# cd /export/servers/
[root@centos6-1 servers]# wget https://archive.apache.org/dist/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz

2.解压安装包

[root@centos6-1 servers]# tar -zvxf apache-storm-1.1.1.tar.gz

3.修改配置文件

这里需要修改的是conf文件夹下的storm.yaml文件，在文件中添加一下内容

#发现zookeeper服务所在服务器
storm.zookeeper.servers:
    - "centos6-1"
    - "centos6-2"
    - "centos6-3"
#配置nimbus服务所在服务器
nimbus.seeds: ["centos6-1", "centos6-2", "centos6-3"]
#storm数据文件存储位置
storm.local.dir: "/export/data/storm/stormdata"
#web访问storm端口
ui.port: 9999
#slots的个数个端口
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

4.分发安装包

能够分发的前提是配置3台服务器主机间的免密登录，方法见博主博文基于SSH的用户名密码验证和免密登录原理

[root@centos6-1 servers]# scp -r apache-storm-1.1.1 centos6-2:$PWD
[root@centos6-1 servers]# scp -r apache-storm-1.1.1 centos6-3:$PWD

5.启动Storm

Storm服务架构

服务	说明
nimbus	负责资源分配和任务调度，3台服务器皆需启动，通过zookeeper形成高可用
supervisor	负责接受nimbus分配的任务，启动和停止属于自己管理的worker进程，3台服务器皆需启动
logViewer	非必须启动，作用是可以通过web访问日志，三台都可启动
ui	非必须启动，可以通过web访问storm的运行状态，这里只需在centos6-1节点启动即可

centos6-1启动nimbus，supervisor，logViewer，ui

[root@centos6-1 servers]# cd apache-storm-1.1.1
[root@centos6-1 apache-storm-1.1.1]# nohup bin/storm nimbus >/dev/null 2>&1 &
[root@centos6-1 apache-storm-1.1.1]# nohup bin/storm supervisor >/dev/null 2>&1 &
[root@centos6-1 apache-storm-1.1.1]# nohup bin/storm logviewer >/dev/null 2>&1 &
[root@centos6-1 apache-storm-1.1.1]# nohup bin/storm ui >/dev/null 2>&1 &

centos6-2启动nimbus，supervisor，logViewer

[root@centos6-2 servers]# cd apache-storm-1.1.1
[root@centos6-2 apache-storm-1.1.1]# nohup bin/storm nimbus >/dev/null 2>&1 &
[root@centos6-2 apache-storm-1.1.1]# nohup bin/storm supervisor >/dev/null 2>&1 &
[root@centos6-2 apache-storm-1.1.1]# nohup bin/storm logviewer >/dev/null 2>&1 &

centos6-3启动nimbus，supervisor，logViewer

[root@centos6-3 servers]# cd apache-storm-1.1.1
[root@centos6-3 apache-storm-1.1.1]# nohup bin/storm nimbus >/dev/null 2>&1 &
[root@centos6-3 apache-storm-1.1.1]# nohup bin/storm supervisor >/dev/null 2>&1 &
[root@centos6-3 apache-storm-1.1.1]# nohup bin/storm logviewer >/dev/null 2>&1 &

6.查看Storm启动状态

查看Storm进程

[root@centos6-1 apache-storm-1.1.1]# jps
75985 logviewer     //logviewer进程
76609 Jps
75984 Supervisor    //supervisor进程
72149 QuorumPeerMain  //zookeeper进程
76404 core     //ui进程
75983 nimbus    //nimbus进程
[root@centos6-2 apache-storm-1.1.1]# jps
65925 Jps
65370 nimbus
65371 Supervisor
65372 logviewer
60767 QuorumPeerMain
[root@centos6-3 apache-storm-1.1.1]# jps
55265 Jps
54710 nimbus
51719 QuorumPeerMain
54711 Supervisor
54712 logviewer

web访问storm UI ： 192.168.214.150:9999
这里写图片描述
在web界面上可以看到Storm的运行正常，Storm启动成功！

7.编写启动关闭脚本

Storm集群的启动需要在每个服务器上单独启动，我们可以通过脚本一键启动和关闭Storm进程。这里不将ui服务作为一键启动的范围内，因为只需一台服务器启动即可，所以ui服务仍需单独启动和关闭

在centos6-1上添加start-storm-all.sh和stop-storm-all.sh和stop-storm.sh3个脚本

[root@centos6-1 apache-storm-1.1.1]# cd bin/
[root@centos6-1 apache-storm-1.1.1]# touch start-storm-all.sh
[root@centos6-1 apache-storm-1.1.1]# touch stop-storm-all.sh
[root@centos6-1 apache-storm-1.1.1]# touch stop-storm.sh

修改权限允许执行

[root@centos6-1 apache-storm-1.1.1]# chmod 777 start-storm-all.sh
[root@centos6-1 apache-storm-1.1.1]# chmod 777 stop-storm-all.sh
[root@centos6-1 apache-storm-1.1.1]# chmod 777 stop-storm.sh

start-storm-all.sh脚本内容

#!/bin/bash
for host in centos6-1 centos6-2 centos6-3
do
ssh $host "source /etc/profile;nohup /export/servers/apache-storm-1.1.1/bin/storm nimbus >/dev/null 2>&1& nohup /export/servers/apache-storm-1.1.1/bin/storm supervisor >/dev/null 2>&1& nohup /export/servers/apache-storm-1.1.1/bin/storm logviewer >/dev/null 2>&1&"
echo "$host storm is starting..."
done

stop-storm-all.sh脚本内容

#!/bin/bash
for host in centos6-1 centos6-2 centos6-3
do
ssh $host "source /etc/profile;nohup /export/servers/apache-storm-1.1.1/bin/stop-storm.sh &" 
echo "$host storm is stopping"
done

stop-storm.sh脚本内容

#!/bin/sh
NIMBUS_PIDS=$(ps ax | grep -i 'nimbus' | grep java | grep -v grep | awk '{print $1}')
SUPERVISOR_PIDS=$(ps ax | grep -i 'Supervisor' | grep java | grep -v grep | awk '{print $1}')
LOGVIEWER_PIDS=$(ps ax | grep -i 'logviewer' | grep java | grep -v grep | awk '{print $1}')

if [ -z "$LOGVIEWER_PIDS" ]; then
  echo "No logviewer server to stop"
else 
  kill -s TERM $LOGVIEWER_PIDS
fi

if [ -z "$SUPERVISOR_PIDS" ]; then
  echo "No supervisor server to stop"
else 
  kill -s TERM $SUPERVISOR_PIDS
fi

if [ -z "$NIMBUS_PIDS" ]; then
  echo "No nimbus server to stop"
  exit 1
else 
  kill -s TERM $NIMBUS_PIDS
fi

将stop-storm.sh脚本分发到centos6-2和centos6-3(如果想在centos6-2和6-3上都能狗一键启动就将其余两个脚本都分发即可)

[root@centos6-1 bin]# scp stop-storm.sh centos6-2:$PWD
[root@centos6-1 bin]# scp stop-storm.sh centos6-3:$PWD

一键关闭Storm

[root@centos6-1 bin]# jps
75985 logviewer         //Storm服务进程都存活
75984 Supervisor
76691 Jps
72149 QuorumPeerMain
76404 core
75983 nimbus
[root@centos6-1 bin]# ./stop-storm-all.sh 
centos6-1 storm is stopping
centos6-2 storm is stopping
centos6-3 storm is stopping
[root@centos6-1 bin]# jps
76757 Jps           //centos6-1的Storm服务都已经被关闭
72149 QuorumPeerMain
76404 core  //core即ui服务，他不在一键启动和关闭的范围内
[root@centos6-2 apache-storm-1.1.1]# jps
66128 Jps       //centos6-2的Storm服务都已经被关闭
60767 QuorumPeerMain
[root@centos6-3 bin]# jps
51719 QuorumPeerMain        //centos6-3的Storm服务都已经被关闭
55469 Jps

一键启动Storm

[root@centos6-1 bin]# ./start-storm-all.sh 
centos6-1 storm is starting...
centos6-2 storm is starting...
centos6-3 storm is starting...

再按照上面的方法查看Storm集群的状态即可

[root@centos6-1 bin]# jps
72149 QuorumPeerMain
76404 core
77108 Jps
76813 nimbus
76815 logviewer
76814 Supervisor
[root@centos6-2 apache-storm-1.1.1]# jps
66449 Jps
66173 nimbus
66175 logviewer
60767 QuorumPeerMain
66174 Supervisor
[root@centos6-3 bin]# jps
55776 Jps
51719 QuorumPeerMain
55514 nimbus
55515 Supervisor
55516 logviewer

这里写图片描述
发现Storm成功启动