storm（五） 消息分组策略

grouping

http://storm.apache.org/releases/1.2.2/Concepts.html

• 消息分发策略决定了消息在spout和bolt间，bolt和bolt间如何进行传递。
• 流分组策略有内置的八种，同时还可以进行自定义。这里将介绍最常用的几种

Shuffle grouping

• 随机分发到两个bolt上，确保两个bolt上tuple的数量大体相同，从日志打印上可看出。

builder.setSpout("createNum",new NumSpout(),1);
builder.setBolt("countNum",new SumBolt(),2)
        .shuffleGrouping("createNum");

Fields grouping

• 按照特定字段来进行分发，保证相同的字段分发到同一个bolt上。
• 配置grouping：

 builder.setSpout("createNum",new NumSpout(),1);
 builder.setBolt("countNum",new SumBolt(),2)
         .fieldsGrouping("createNum",new Fields("isOdd"));//按照isOdd进行分区

• 由于是按照特定的field进行分发，所以需要在额外添加一个isOdd字段用来分发：

public class NumSpout extends BaseRichSpout{
    private SpoutOutputCollector collector;
    private Integer number=0;
    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        this.collector=collector;
    }
    public void nextTuple() {
        int num=++number;
        collector.emit(Arrays.asList((Object)num,num%3));
        System.out.println("number:"+number);
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("num","isOdd"));//声明发出的字段分别是num和isOdd
    }
}

All Grouping

• 数据会传给所有的bolt上，换句话说数据可能会被重复执行

 builder.setSpout("createNum",new NumSpout(),1);
 builder.setBolt("countNum",new SumBolt(),2)
         .allGrouping("createNum");

direct grouping

https://blog.csdn.net/simon_09010817/article/details/81364357

• 将数据定向发送到一个指定的bolt上（由发送者决定目标bolt）

public class NumSpout extends BaseRichSpout{

    private SpoutOutputCollector collector;
    private Integer number=0;
    private List<Integer> idList;

    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        this.collector=collector;
        //根据componentId(bolt或者是spout)获取所有的task id
        this.idList=context.getComponentTasks("countNum");
    }

    public void nextTuple() {
        int num=++number;
        //将消息发送到指定的task id上
        collector.emitDirect(idList.get(1),new Values(num));
        System.out.println("number:"+number);

        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("num"));
    }
}

builder.setSpout("createNum",new NumSpout(),1);
builder.setBolt("countNum",new SumBolt(),3)
        .directGrouping("createNum");