JStorm学习笔记-集群环境安装部署

安装版本如下：

jstorm-2.2.1

zookeeper-3.4.10

zeromq-4.1.5

先行启动zookeeper的集群服务

修改配置文件conf/storm.yaml内容

########### These MUST be filled in for a storm configuration
 storm.zookeeper.servers:
     - "192.168.0.11"
     - "192.168.0.12"
     - "192.168.0.13"

 storm.zookeeper.root: "/jstorm"

 cluster.name: "jstorm-cluster"

 #nimbus.host/nimbus.host.start.supervisor is being used by $JSTORM_HOME/bin/start.sh
 #it only support IP, please don't set hostname
 # For example
 nimbus.host: "192.168.0.11"
 #nimbus.host: "localhost"
 nimbus.host.start.supervisor: true
 
# %JSTORM_HOME% is the jstorm home directory
 storm.local.dir: "/home/data/jstorm-2.2.1/data"
 # please set absolute path, default path is JSTORM_HOME/logs
 jstorm.log.dir: "/home/data/jstorm-2.2.1/logs"
 
 java.library.path: "/usr/local/jdk1.8.0_102/lib:/usr/local/lib:/home/data/zeromq-4.1.5:/home/data/jstorm-2.2.1/extlib"

# if supervisor.slots.ports is null, 
# the port list will be generated by cpu cores and system memory size 
# for example, 
# there are cpu_num = system_physical_cpu_num/supervisor.slots.port.cpu.weight
# there are mem_num = system_physical_memory_size/(worker.memory.size * supervisor.slots.port.mem.weight) 
# The final port number is min(cpu_num, mem_num)
# supervisor.slots.ports.base: 6800
# supervisor.slots.port.cpu.weight: 1.2
# supervisor.slots.port.mem.weight: 0.7
# supervisor.slots.ports: null
 supervisor.slots.ports:
    - 6800
    - 6801
    - 6802
    - 6803

# Default disable user-define classloader
# If there are jar conflict between jstorm and application, 
# please enable it 
 topology.enable.classloader: false

# enable supervisor use cgroup to make resource isolation
# Before enable it, you should make sure:
# 	1. Linux version (>= 2.6.18)
# 	2. Have installed cgroup (check the file's existence:/proc/cgroups)
#	3. You should start your supervisor on root
# You can get more about cgroup:
#   http://t.cn/8s7nexU
 supervisor.enable.cgroup: false


## send message with sync or async mode
## if this setting is true, netty will use sync mode which means client can send one batch message only after receive one server's response
## Async mode means client can send message without server's response
 storm.messaging.netty.sync.mode: false

### Netty will send multiple messages in one batch  
### Setting true will improve throughput, but more latency
 storm.messaging.netty.transfer.async.batch: true
 
### default worker memory size, unit is byte
 worker.memory.size: 2147483648

# Metrics Monitor
# topology.performance.metrics: it is the switch flag for performance 
# purpose. When it is disabled, the data of timer and histogram metrics 
# will not be collected.
# topology.alimonitor.metrics.post: If it is disable, metrics data
# will only be printed to log. If it is enabled, the metrics data will be
# posted to alimonitor besides printing to log.
 topology.performance.metrics: true
 topology.alimonitor.metrics.post: false

# when supervisor is shutdown, automatically shutdown worker
# if run jstorm under other container such as hadoop-yarn, 
# this must be set as true
 worker.stop.without.supervisor: false

#set how many tuple can spout send in one time.
# For example, if this is setting 100, 
# spout can't send the No. 101th tuple until spout receive one tuple's ack message
# topology.max.spout.pending: null

# UI MultiCluster
# Following is an example of multicluster UI configuration
# ui.clusters:
#     - {
#         name: "jstorm",
#         zkRoot: "/jstorm",
#         zkServers:
#             [ "localhost"],
#         zkPort: 2181,
#       }

mkdir ~/.jstorm

cp -f /home/data/jstorm-2.1.1/conf/storm.yaml ~/.jstorm

拷贝目录到集群各个节点，在NimbusServer节点上将jstorm-ui-2.2.1.war拷贝到tomcat指定目录下

编辑脚本 startup.sh、shutdown.sh

#!/bin/bash
/home/data/jstorm-2.1.1/bin/jstorm nimbus &
/home/data/jstorm-2.1.1/bin/jstorm supervisor &
/home/data/jstorm-2.1.1/webui/bin/startup.sh

#!/bin/bash
jps|grep NimbusServer|awk '{print "kill -9 " $1}'|sh
jps|grep Supervisor|awk '{print "kill -9 " $1}'|sh
/home/data/jstorm-2.1.1/webui/bin/shutdown.sh

启动成功后可以通过http://192.168.0.11:16666/jstorm-ui-2.2.1 查看集群和拓扑的信息

用例可以在本地模式环境下直接运行测试，也可以打成jar包在集群环境中通过运行

bin/jstorm jar example.jar main类参数1 参数2 参数3 进行测试

jstorm命令

jstorm command [--config client_storm.yaml] [--exclude-jars exclude1.jar,exclude2.jar] [-c key1=value1,key2=value2][command parameter]
Commands:
    activate
    blacklist
    blobstore
    classpath
    deactivate
    drpc
    help
    jar
    kill
    list
    localconfvalue
    metricsMonitor
    nimbus
    rebalance
    remoteconfvalue
    restart
    supervisor
    update_topology
    zktool

    [--config client_storm.yaml]             optional, setting client's storm.yaml

    [--exclude-jars exclude1.jar,exclude2.jar]     optional, exclude jars, avoid jar conflict

    [-c key1=value1,key2=value2]             optional, add key=value pair to configuration

支持添加第三方依赖jar包，在JStorm目录下新建目录extlib，编辑bin/jstorm，修改内容

def get_classpath(extrajars):
    ret = []
    ret.extend(extrajars)
    ret.extend(get_jars_full(JSTORM_DIR))
    ret.extend(get_jars_full(JSTORM_DIR + "/lib"))
    ret.extend(get_jars_full(JSTORM_DIR + "/extlib"))
    ret.extend(INCLUDE_JARS)
    return normclasspath(":".join(ret))

第三方依赖jar包放入extlib目录下重启服务即可。

JStorm学习笔记-集群环境安装部署

猜你喜欢