File configuration:
storm.zookeeper.servers:
- "data1"
- "data2"
storm.zookeeper.port: 2181
storm.local.dir: "/mnt/storm"
nimbus.seeds: ["data2","data3"]
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
Operation command:
Background start script command:
1. Nohup
example: nohup ./storm nimbus &
Explanation: start a script command in the background, and output the log generated by the script to the nohup.out file in the same directory as the script.
2. >/dev/null 2>&1 &
Example: ./storm nimbus >/dev/null 2>&1 &
Explanation: Start a script command in the background and discard any output log information generated by the script command.
Storm common commands can visit the official website:
http://storm.apache.org/releases/1.0.2/Command-line-client.html
1. nimbus
example: storm nimbus
explanation: start nimbus (main) daemon
2. supervisor
example : storm supervisor
explain: start supervisor (worker node) daemon
3. ui
example: storm ui
explain: start ui daemon thread. Address http://data2:8080/index.html
4. jar
example: storm jar /opt/stromdemo-1.0-SNAPSHOT-jar-with-dependencies.jar com.cetc.stromdemo.wordcount.WordCountTopology '/word_count.txt'
Explanation: Submit a topology to the storm cluster
5. kill
Example: storm kill WordCountTopology
Explanation: Kill a running topology
6. deactivate
Example: storm deactivate WordCountTopology
Explanation: Disable a running topology
7. activate
Example: storm activate WordCountTopology
Explanation: Activates a disabled topology
8. rebalance
Example: storm rebalance WordCountTopology -w 5 -n 4 -e LineSplitBolt=1
Explanation: For an already running task, re-change the number of workers and the bolt or spout concurrency. However, when changing the degree of concurrency, it can only be reduced, and it is ineffective to increase the degree of concurrency. storm rebalance topology-name [-w wait-time-secs] [-n new-num-workers] [-e component=parallelism]*
9. repl
example: storm repl
explain: open clojure command line operations where you can Use lisp
10. classpath
For example: storm classpath
Explanation: View the classpath included by storm in this machine
11. localconfvalue
For example: storm localconfvalue storm.zookeeper.servers
Explanation: Output a configured value of Storm in the local configuration file according to the key
12.
Remoteconfvalue Example: storm remoteconfvalue storm.zookeeper.servers
Explanation: Output the configuration value of Storm in the cluster according to the key
13 .list
For example: storm list
Explanation: List the running topology and its status.
14. version
For example: storm version
Explanation: Check the storm version
Code example:
import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;
import org.apache.storm.utils.Utils;
import java.util.Map;
import java.util.Random;
/**
* Created by shea on 2018/2/2.
*/
public class RandomNameSpout extends BaseRichSpout {
private static final long serialVersionUID = 1L;
// output collector
private SpoutOutputCollector collector;
// simulate some data
String[] names = { "zhangsan", "lisi", "wangwu", "zhaoliu", "sunqi", "wangba" };
/**
* Continuously send tuple messages to the next component This is the core logic of the spout component
*/
public void nextTuple() {
Random random = new Random();
int index = random.nextInt(names.length);
// get a name by random number
String lowerName = names[index];
// Encapsulate the name into a tuple and send a message to the next component
collector.emit(new Values(lowerName));
// Every time a message is sent, sleep for 500ms
Utils.sleep(500);
}
/**
* The initialization method, which is called once when the spout component is instantiated
*/
public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
this.collector = collector;
}
/**
* Declare the field name of the data in the tuple sent by this spout component
*/
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("lowerName"));
}
}
import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
import java.util.Map;
/**
* Created by shea on 2018/2/2.
*/
public class UpperBolt extends BaseRichBolt {
private static final long serialVersionUID = 1L;
private OutputCollector collector;
//retrieve data
public void execute(Tuple tuple) {
// First get the data passed by the previous component, the data is in the tuple
String lowerName = tuple.getString(0);
// Convert name to uppercase
String upperName = lowerName.toUpperCase();
// Send the converted product name
collector.emit(new Values(upperName));
}
public void prepare(Map conf, TopologyContext topologyContext, OutputCollector collector) {
this.collector = collector;
}
/**
* Declare the field of the tuple that the bolt component will send out
*/
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("upperName"));
}
}
import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Tuple;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Map;
import java.util.UUID;
/**
* Created by shea on 2018/2/2.
*/
public class AppendBolt extends BaseRichBolt {
private static final long serialVersionUID = 1L;
FileWriter fileWriter = null;
public void execute(Tuple tuple) {
// First get the name sent by the previous component
String upperName = tuple.getString(0);
String suffix_name = upperName + "_csdn";
try {
fileWriter.write(suffix_name);
fileWriter.write("\n");
fileWriter.flush();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
/**
* will only be called once during the operation of the bolt component
*/
public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
try {
//fileWriter = new FileWriter("C:\\Users\\91BGJK2\\Desktop\\storm\\" + UUID.randomUUID());
fileWriter = new FileWriter("/mnt/" + UUID.randomUUID());
} catch (IOException e) {
throw new RuntimeException(e);
}
}
/**
* This bolt no longer needs to send a tuple message to the next component, so there is no need to declare the tuple field
*/
public void declareOutputFields(OutputFieldsDeclarer arg0) {
}
import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
import org.apache.storm.StormSubmitter;
import org.apache.storm.generated.StormTopology;
import org.apache.storm.topology.TopologyBuilder;
/**
* storm's test program
* Created by shea on 2018/2/2.
*/
public class StormMain {
public static void main(String[] args) throws Exception {
//topology constructor
TopologyBuilder builder = new TopologyBuilder();
// Set our spout component to topology
// parallelism_hint: 4 means that 4 excutors are used to execute this component ----- should mean 4 threads
// setNumTasks(8) sets the number of concurrent tasks when the component is executed, which means that one executor will run two tasks
builder.setSpout("randomspout", new RandomNameSpout(), 4).setNumTasks(8);
// Set the uppercase conversion bolt component to the topology and specify that it receives messages from the randomspout component
// .shuffleGrouping("randomspout") contains two meanings:
// 1. The tuple message received by the upperbolt component must come from the randomspout component
// 2. Massive concurrency of randomspout components and upperbolt components
// The grouping strategy used when sending and receiving messages between task instances is random grouping shuffleGrouping
builder.setBolt("upperbolt", new UpperBolt(), 4).shuffleGrouping("randomspout");
// Set the suffixed bolt component to the topology and specify that it receives messages from the upperbolt component
builder.setBolt("suffixbolt", new AppendBolt(), 4).shuffleGrouping("upperbolt");
// use the builder to create a topology
StormTopology stormTopologyDemo = builder.createTopology();
// Configure some parameters when the topology runs in the cluster
Config conf = new Config();
// The number of slots occupied by the entire StormTopologyDemo is set here, that is, the number of workers
conf.setNumWorkers (4);
conf.setDebug(true);
conf.setNumAckers (0);
LocalCluster cluster = new LocalCluster();
//cluster.submitTopology("StormTopologyDemo",conf,stormTopologyDemo);
// Submit this topology to the storm cluster to run
StormSubmitter.submitTopology("StormTopologyDemo", conf, stormTopologyDemo);
}
}
Stand-alone startup:
It can be run directly in the program (call cluster.submitTopology("StormTopologyDemo",conf,stormTopologyDemo);), if it is run in a cluster, it needs to be packaged on the cluster and run with the command
./storm jar /mnt/storm-1.0-SNAPSHOT.jar StormMain