Part 1: Introduction to Storm's cluster mode:
sent by the Nimbus master node Introduction to the
construction
process: 1. Stand-alone version
Environment
Java 6+
Python 2.6.6+ // python -V View the version
Construction
1. Unzip the storm archive and create a new logs file in it.
2. Get help execution by viewing help../bin/storm help
3. Start zk and set the log output:./bin/storm dev-zookeeper >> ./logs/dev-zookeeper.out 2>&1 &
Explanation : 2 means error information & 1 is output to zookeeper.out. The & at the end means that it runs in the background. And through jps to observe the process operation, first config_value, and then run zookeeper.
4. Start numbus:./bin/storm nimbus >> ./logs/nimbus.out 2>&1 &observe through jps.
5. Start the supervisor:./bin/storm supervisor >> ./logs/supervisor.out 2>&1 &observe through jps.
6. Start UI./bin/storm ui>> ./logs/ui.out 2>&1 &
7. Any,./bin/storm logviewer >> ./logs/logviewer.out 2>&1 &
9. View its port:ss -nal
8. Test access through web page:node01:8080
9. After the jar of the generated test case wordCount is uploaded to the cluster,
./storm jar ~/testDemo/storm_wc.jar com.jw.storm.test.Test_cluster xxx
2. Fully distributed (node02, node03, node04):
1. All servers in the cluster, synchronize all configurations! (Distribution)
Specific configuration: Modify storm.zookeeper.servers: node02, node03, node04
and nimbus.host: “node02” storm.local.dir: “/tmp/storm”
and finally configure the supervisor port.
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
2. Start the ZooKeeper cluster
3. Start Nimbus on node2
Startup: Start nimbus and ui on the master node of node02. Start supervisor on node03 and node04.
node02:
./bin/storm nimbus >> ./logs/nimbus.out 2>&1 &
tail -f logs/nimbus.log
./bin/storm ui >> ./logs/ui.out 2>&1 &
tail -f logs/ui.log
node03:
Nodes node3 and node4 start the supervisor. According to the configuration, there are 4 slots for each supervisor started.
./bin/storm supervisor >> ./ logs/supervisor.out 2>&1 &
tail -f logs/supervisor.log
(of course, node2 can also start the supervisor)
After all the startups are completed, start the zookeeper client (zkCli.sh) of one of the nodes. View the directory of zk through ls /. A directory with storm can be found. Inside is the process and heartbeat and allocation information of the communication between nimbus and zk. As follows:
[zk: localhost:2181(CONNECTED) 5] ls /storm
[workerbeats, storms, supervisors, errors, assignments]
Part II: Introduction to Storm's concurrency mechanism:
Worker processes
Executors (threads)
Tasks
1. In a Storm cluster, for a topology, a node can run one or more processes. Each process can run multiple threads.
2. Each process can run one or more threads, and each thread can run multiple tasks.
3. Each task performs specific data processing.
In summary:
Worker – Process
A Topology topology will contain one or more Workers (each Worker process can only belong to a specific Topology)
These Worker processes will run in parallel on different servers in the cluster, that is, a Topology topology In fact, it is composed of processes running in parallel on multiple servers in the Storm cluster.
Executor – A thread
Executor is a thread generated by a Worker process.
Each Worker process will run one or more Executor threads
in the topology. In an Executor thread One or more Task tasks can be executed (by default each Executor only executes one Task task), but these Task tasks all correspond to the same component (Spout, Bolt). The smallest unit of data processing actually performed by a task. Each
task is a Spout or a Bolt . The number of tasks remains unchanged throughout the topology life cycle, and the number of executors can be changed or manually adjusted (by default, the number of tasks and executors are the same, That is, a Task task runs by default in each Executor thread) Description of parameter settings: Set the number of Worker processes Config.setNumWorkers(int workers)
Set the number of Executor threads
TopologyBuilder.setSpout(String id, IRichSpout spout, Number parallelism_hint)
TopologyBuilder.setBolt(String id, IRichBolt bolt, Number parallelism_hint)
: where parallelism_hint is the number of executor threads
Set the number of
Tasks ComponentConfigurationDeclarer.setNumTasks(Number val)
例：
Config conf = new Config() ;
conf.setNumWorkers(2);
TopologyBuilder topologyBuilder = new TopologyBuilder();
topologyBuilder.setSpout("spout", new MySpout(), 1);
topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)
.setNumTasks(4)
.shuffleGrouping(" blue-spout);
two adjustment methods:
1. Command line control:
./bin/storm rebalance wc -e WCountbolt=4
standby: (full version)storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
Description: Adjust the number of mytopology topology worker processes to 5 "blue-spout" and adjust the number of threads used by 3
"yellow-bolt" Adjusted the number of threads used to 10
Then view the adjusted thread distribution through the Storm UI interface. (Note that the total number of threads will be more than the total number of operation settings. There are threads monitored by acker, and the specific number is determined by the number of bolts set)
2. Directly through the rebalance control of Storm UI.
Storm communication mechanism:
The original method was ZMQ invented by twitter, which was later replaced by Apache's Netty.
Data communication between worker processes
ZMQ
ZeroMQ is an open source messaging framework, not a MessageQueue
Netty
Netty is a NIO-based network framework, which is more efficient. (The reason why Netty is used after Storm 0.9 is because the license of ZMQ is not compatible with the license of Storm.)
The data communication
Disruptor inside the Worker
implements the function of "queue".
It can be understood as an event monitoring or message processing mechanism, that is, on one side of the queue, the producer puts message data, and on the other side, the consumer takes out the message data for processing in parallel.
Part 3: Introduction to Storm's fault tolerance mechanism:
First understand a question, why doesn't Nimbus do high availability? Don't single nodes have the possibility of a single point of failure? Of course, Nimbus plays the role of task allocation, resource control, and jar uploading in the entire cluster. It does not want the inseparable relationship between nameNode and dataNode, and metadata information is stored on NN. And Nimbus saves the following supervisor information in zookeeper. When Nimbus hangs, the relationship between zookeeper and supervisor is not interrupted. There's nothing wrong with assigning tasks and waiting for them to be completed. Although it is the master node, the task is relatively light, and it does not have a big impact on the cluster in a short time. But when one of the jobs hangs up, zookeeper will notify Nimbus, and it will assign other jobs to work on this work's tasks. Or NImbus hangs when pushing the jar. But without it does not immediately affect the operation of the entire cluster. Can be started manually.
Secondly, the single point problem of zookeeper, which can be made highly available. Finally, there is the high availability of the supervisor, which is highly available. When one of the nodes goes down, the heartbeat with zookeeper will be interrupted, and then zookeeper will notify Nimbus, and Nimbus will redistribute work tasks to other supervisors through zookeeper. When the failed supervisor is alive, it will remember suicide. way to kill all the work that was running on you before. It's called an automatic destruction mechanism. Prevent repeated operations.
Finally, there is the problem of work hanging. When the work hangs, zookeeper will not be able to check its heartbeat. First let the supervisor try to restart the work. If the zookeeper fails, the task on this work is assigned to the supervisor of other nodes to do it. Then the work will also have the process of automatic destruction.
2 important API methods:
the completion of the message (ack flag information completion successfully, fail flag information completion failure)
Introduction: The tuple sent from Spout, and the tuple based on it (such as the sentence sent by Spout in the previous example, and the tuple of words in the sentence, etc.)
form a tuple tree
when the tuple tree is sent. , and each message in the tree is processed correctly, it means that the message sent by the spout is "completely processed", that is, the integrity of the message.
Understanding of ack:
Acker - Mechanism for implementing message integrity.
Some special tasks in Storm's topology are
responsible for tracking the DAG (directed acyclic graph) of Tuples sent by each Spout. You
need to provide messageId to track whether the execution is successful.
The message protection mechanism, the ack method is called if it succeeds, and the fail method is called if it fails.
The fourth part: Introduction to Storm's communication mechanism:
DRPC: Distributed Remote Call
It
The client requests, the DRPC service accepts the request, and then sends the request to storm. The result calculated by it is not sent directly to the client, but is sent through the DRPC service.
DRPC design purpose:
to quickly return to the client for the settlement of big data.
In order to make full use of Storm's computing power to achieve high-density parallel real-time computing.
(Storm receives several data stream inputs, the data is run in the Topology, and then the results are output through DRPC.)
The diagram is as follows:
The requestId here means that when multiple clients request at the same time, they can be returned accurately.
The lightning bolt icon above is resultBolt.
Storm's operating mode:
1. Local mode
TopologyBuilder builder = new TopologyBuilder();
LocalDRPC drpc = new LocalDRPC();
DRPCSpout spout = new DRPCSpout("exclamation", drpc);
builder.setSpout("drpc", spout);
builder.setBolt("exclaim", new ExclamationBolt(), 3).shuffleGrouping("drpc");
builder.setBolt("return", new ReturnResults(), 3).shuffleGrouping("exclaim");
LocalCluster cluster = new LocalCluster();
Config conf = new Config();
cluster.submitTopology("exclaim", conf, builder.createTopology());
System.err.println(drpc.execute("exclamation", "aaa"));
System.err.println(drpc.execute("exclamation", "bbb"));2. Cluster mode (remote mode)
modify the configuration file conf/storm.yaml
drpc.servers:
- "node1"
start DRPC Server
bin/storm drpc &
submit topology through StormSubmitter.submitTopology