Storm cluster deployment

JStorm cluster deployment

ZooKeeper high availability cluster installation and configurationhttp
://wosyingjun.iteye.com/blog/2312960

1. Download and upload zookeeper-3.4.6.tar.gz to the /usr/local/ directory of each server
$ cd /usr/local/
$ wget http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

2. Unzip the zookeeper installation package on each server, and pair it by node number zookeeper directory rename (you can not rename the directory)
$ tar -zxvf zookeeper-3.4.6.tar.gz
server 1:
$ mv zookeeper-3.4.6 zookeeper-3.4.6_(1)
server 2:
$ mv zookeeper-3.4 .6 zookeeper-3.4.6_(2)
Server 3:
$ mv zookeeper-3.4.6 zookeeper-3.4.6_(3)

3. Create the following directory under each zookeeper node directory:
$ cd /usr/local/zookeeper-3.4 .6_(x) (x represents the node number)
$ mkdir data
$ mkdir logs

4. Copy the zoo_sample.cfg file in the zookeeper/zookeeper-3.4.6_(x)/conf directory and name it zoo.cfg:
$ cp zoo_sample.cfg zoo.cfg

5. Modify the zoo.cfg configuration file

#zookeeper-3.4.6_(1) configuration (/usr/local/zookeeper-3.4.6_(1)/conf/zoo.cfg) as follows:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper-3.4.6_(1)/data
dataLogDir=/usr/local/zookeeper-3.4.6_(1)/logs
clientPort=2181
server.1= 192.168.11.97:2881:3881
server.2=192.168.11.98:2882:3882
server.3=192.168.11.99:2883:3883

#zookeeper-3.4.6_(2) configuration (/usr/local/zookeeper-3.4. 6_(2)/conf/zoo.cfg) as follows:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper-3.4.6_(2)/data
dataLogDir=/usr/local/zookeeper-3.4. 6_(2)/logs
clientPort=2182
server.1=192.168.11.97:2881:3881
server.2=192.168.11.98:2882:3882
server.3=192.168.11.99:2883:3883

#zookeeper-3.4.6_(3) configuration (/usr/local/zookeeper-3.4.6_(3)/conf/ zoo.cfg) as follows:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper-3.4.6_(3)/data
dataLogDir=/usr/local/zookeeper-3.4.6_(3)/logs
clientPort =2183
server.1=192.168.11.97:2881:3881
server.2=192.168.11.98:2882:3882
server.3=192.168.11.99:2883:3883

6. In dataDir=/usr/local/zookeeper-3.4.6_ (x) Create a myid file under /data
$ vi /usr/local/zookeeper-3.4.6(1) /data/myid Set the value of the file content to 1
$ vi /usr/local/zookeeper-3.4.6(2) /data/myid sets the value of the file content to 2
$ vi /usr/local/zookeeper-3.4.6_(3) /data/myid sets the value of the file content to 3

7. Open the ports 218X, 288X, 388X to be used in the firewall (X represents a node, such as node 1, the ports to be opened are 2181, 2881, 3881)
$ vi /etc/sysconfig/iptables
-A INPUT - m state --state NEW -m tcp -p tcp --dport 218X -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 288X -j ACCEPT
-A INPUT -m state -- state NEW -m tcp -p tcp --dport 388X -j ACCEPT
$ service iptables restart
or close the firewall #close
the firewall
/etc/init.d/iptables stop
service iptables stop # stop service #view
firewall information
/etc/init.d /iptables status

8. Start and view zookeeper:
$ /usr/local/zookeeper-3.4.6(x)/bin/zkServer.sh start
$ /usr/local/zookeeper-3.4.6(x)/bin/zkServer. sh status

9. Modify the client configuration for connecting to zookeeper:
zookeeper://192.168.11.97:2181?backup=192.168.11.98:2182,192.168.11.99:2183


Installation and use of JStorm2.1.1 cluster
http://wosyingjun.iteye.com/blog/2317034
http://wosyingjun. iteye.com/blog/2316763

First create an admin user in CentOS

1. For the installation of Zookeeper cluster,
see above: Installation and configuration of ZooKeeper high availability cluster

2. Installation of JStorm (JDK1.8 and Python2.6 need to be installed before installation Above)
download JStorm
wget https://github.com/alibaba/jstorm/wiki/Downloads

on server A and server B: (Add the following content to .bash_profile and .bashrc in the home directory of the admin user)
unzip jstorm- 2.1.1 Add the following content to
vim /etc/profile
#profile:
export JSTORM_HOME=/XXXXX/XXXX
export PATH=$PATH:$JSTORM_HOME/bin

Configure on server A:
 vim /home/jstorm/jstorm-2.1.1/ conf/storm.yaml
storm.zookeeper.servers:
     - "192.168.xx.100"
     - "192.168.xx.101"
     - "192.168.xx.102"
 nimbus.host: "192.168.xx.100"
 storm.zookeeper.root: "/jstorm"
 storm.local.dir : "%JSTORM_HOME%/data"
 supervisor.slots.ports:
    - 6800
    - 6801
    - 6802
    - 6803

nimbus.childopts: "-Xmx1024m"

storm.zookeeper.servers: The
host name is required, for example:
storm.zookeeper.servers:
     - "storm-1"
     - "storm-2"
     - "storm-3"
correspond to three hosts respectively, each host must add
192.168.xx.100 storm-1
192.168.xx.101 storm to /etc/hosts -2
192.168.xx.102 storm-3

nimbus.host:This configuration item can only be configured

on server B with an ip address, not a hostname:
 vim /home/jstorm/jstorm-2.1.1/conf/storm.yaml
storm.zookeeper.servers:
     - "192.168.xx.100"
     - "192.168.xx.101"
     - "192.168.xx.102"
 nimbus.host : "192.168.xx.100"
 storm.zookeeper.root: "/jstorm"
 storm.local.dir: "%JSTORM_HOME%/data"
 supervisor.slots.ports:
    - 6800
    - 6801
    - 6802
    - 6803

nimbus.childopts: " -Xmx1024m"

storm.zookeeper.servers: The
host name is required, for example:
storm.zookeeper.servers:
     - "storm-1"
     - "storm-2"
     - "storm-3"
corresponds to three hosts, each host The /etc/hosts must be added with
192.168.xx.100 storm-1
192.168.xx.101 storm-2
192.168.xx.102 storm-3




nimbus.host : This configuration
item can only use ip address, not hostname The content of storm.yaml remains the same.
mkdir ~/.jstorm
cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm


4. Start JStorm and start it
on server A:
nohup jstorm nimbus &
nohup jstorm supervisor & #Check the
startup situation
jps start
on server B:
nohup jstorm supervisor & #Check the
startup situation
Jps

deploy TOPOLOGY on server A
jstorm jar Leek-1.0-SNAPSHOT.jar com.yingjun.stock.topology.StockStategyTopology conf.yaml

If there are other yaml files in the directory where conf.yaml is located, it will cause TOPO Startup failed.
Note: For the writing method of TOPOLOGY, please refer to the project: Leek - Simple version of real-time intelligent stock selection platform


5. Install JStorm Web UI and use the war package mkdir ~/.jstorm
of jstorm-ui-2.1.1

cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm #Download
tomcat 7.x (take apache-tomcat-7.0.37 as an example)
tar -xzf apache-tomcat-7.0.37.tar.gz
cd apache- tomcat-7.0.37
cd webapps
cp $JSTORM_HOME/jstorm-ui-0.9.6.3.war ./
​​../bin./startup.sh
Access address: http://192.168.xx.xxx:8080/jstorm-ui- 2.1.1/