The installation of a Storm cluster is divided into the following steps:
1. First ensure the normal operation of the Zookeeper cluster service and the correct installation of the necessary components
2. Release the compressed package
3. Modify storm.yaml to add cluster configuration information
4. Use the storm script to start the corresponding service and view the service status
5. View the status of the Storm cluster through the web
Before installing Storm, first ensure that the previously installed Zookeeper service is running normally, including configuring hosts mapping, modifying host names, and firewalls have been set up.
Storm is written in java, so it must rely on JDK to run, the system should first install JDK correctly
Some of them need to rely on Python. The default Python version of Red Hat series Linux is 2.6.6, which can meet the requirements; Linux can install multiple versions of Python to coexist. It is recommended that the Python version be 2.7.x during the production process.
The host names of the three virtual machines used in this test are: hadoopha, hadoop1, hadoop2
First go to the Apache Storm official website to download the installation package, the URL is: http://storm.apache.org/, after entering, click the DOWNLOAD button above to enter the download list
The version of storm 0.9.5 is downloaded here. After downloading, upload it to the server directory, release it and put it in the specified directory:
$ tar -xvzf apache-storm- 0.9 . 5 . tar .gz $ mv apache-storm-0.9.5 /usr/ $ cd /usr/apache-storm-0.9.5
Next, you need to modify the configuration file storm.yaml and execute vim conf/storm.yaml to open the file:
Remove the previous comment of storm.zookeeper.servers: and modify it to all hosts in the cluster that deploy zookeeper. Of course, you can add them manually. The specific configuration is as follows:
Add storm.local.dir option, specify nimbus, supervisor process is used to store a small amount of state data, such as jar package, configuration file, etc.
After writing the configuration file, we need to manually create this directory
The following specifies the supervisor worker node. It is necessary to configure the number of workers that the node can run. Each worker occupies one port for receiving messages, and a maximum of 5 are allocated; Ports 6702 and 6703, three ports are defined here, which represent up to three workers running:
The cluster host is set below so that all nodes in the cluster can download topology and configuration files from the host. The host runs nimbus, and other nodes are supervisor processes. Here hadoopha is nimbus, and hadoop1 and hadoop2 are supervisors, so the configuration is as follows:
Configure the drpc address of the Storm cluster below, here is hadoopha, you can actually define it yourself:
Finally, configure the allocated memory of the storm process. By default, when Storm starts the worker process, the maximum memory of the JVM is 768M. Since a large amount of data is loaded in the Bolt during use, the 768M memory cannot meet the requirements, which will lead to memory overflow. Modify the situation, here is set to 2G
The above settings are fine, save the configuration file and exit
Then create the data directories set above on the 3 hosts respectively, all of which must be created:
mkdir -p /usr/data/storm
The above configuration is configured on hadoopha, the next step is to send the storm directory to hadoop1 and hadoop2:
$ scp -r apache-storm-0.9.5 hadoop1:/usr/ $ scp -r apache-storm-0.9.5 hadoop2:/usr/
发送之后，进入storm安装目录，开始启动相应服务
首先启动Nimbus服务，只在hadoopha上执行：
nohup bin/storm nimbus >> /dev/null &
上面命令的意思是丢弃输出信息并且放到后台执行，稍微等一下，执行jps查看nimbus进程是否启动：
然后在hadoop1，hadoop2节点都启动Supervisor服务：
nohup bin/storm supervisor >> /dev/null &
稍等一下，也可以用jps查看到supervisor进程，
然后在配置drpc的主机hadoopha，drpc是一种后台服务，用于执行和storm相同的计算，但是比较节省资源，一般和nimbus使用同一台主机即可；执行以下命令启动drpc服务：
nohup bin/storm drpc >> /dev/null &
稍等一下可以分别通过jps命令查看到drpc进程，
最后在nimbus节点也就是hadoopha执行以下命令，启动UI服务：
nohup bin/storm ui >> /dev/null &
通过jps可以查看core进程是否启动，启动之后通过访问hadoopha的ip即可进入web管理界面：http://192.168.1.42:8080
到这里基本的storm集群就配置完毕了