Hotspot detection - what HotKey can do in IoT

I. Introduction

        Today, the blogger will talk about hotspot detection in the Internet of Things field, and talk about the architecture and source code. The middleware relied on is mainly JD Wu Weifeng’s HotKey (here has been approved by the creator) hotkey: JD App background middleware, millisecond-level detection Hot data is pushed to the memory of the server cluster in milliseconds, which greatly reduces the pressure of hot keys on the data layer query         .

        It sounds a bit official. Let’s take a look at the scenarios where this middleware can be used in the Internet of Things and asset fields, and then introduce its implementation principle, which is convenient for construction and secondary development.

2. IoT and asset scenarios

1. Jump

        The machine will cause jumps in some specific use methods and combinations. In fact, there are many generalized contents, mainly because of repeated and continuous actions, such as loosening and tightening the bayonet, and opening and closing the travel switch repeatedly.

        These will lead to frequent and invalid processing of business scenarios such as early warning and repair, resulting in the consumption of a large number of IoT heartbeats, wasting performance and resources.

        In fact, HotKey can be used here. For example, if the rule is set to receive more than five transitions in two seconds, it will be marked as a problem, stop useless processing, and after identification, it will be handed over to the hardware for identification and detection.

        Some students may say, can I do it myself in the service?

        Yes, but this is biased towards technical requirements, and the problems caused are not serious. It is not realistic to spend time and effort to make a somewhat customized rather than general requirement. But after using the framework, in addition to the cost of construction and operation and maintenance, everything else is common, and as a service developer, you don't care about these operations and maintenance, and it is easy to use.

2. Query timeout

        Performance is very important for invoking other people's services or calling your own services for others. For systems with low performance requirements, such as store lists, machine lists, and related organization areas, second-level queries are definitely just needed. , otherwise the service quality management team will come to you. A large part of the queries are unsolvable because:

        1. Excessive calls to external systems

        2. When the external system is called or the external query is provided, the traffic suddenly peaks, and then the IO of the disk or the network instantly goes up.

        In this case, hotkey can also be used to temporarily cache hot data and query it directly from local memory, and then SLA will not cause trouble.

3. Principle

1. Hot key detection

1.1. Detection strategy

        Many people in the open source community thought that its strategy was based on interception when they didn't understand this middleware at the beginning, such as intercepting the request sent to redis and sending it to the server to calculate whether it is a hot key.

        In fact, this is not the case, and it is aimed at all hot keys, that is to say, it does not distinguish what third-party storage is, whether it is es, db, redis, etc. third-party storage, it can calculate hot keys.

        So how did he do it? In fact, he considered it based on business thinking. His creator, Wu Weifeng, is also a business developer, not a middleware team.

        So what is the business thinking? If you need to expose or count a key-value to the outside world, then you will be checked first to see if it is a hot key. If it is, you will get the data from the local cache. At this time, it means that the key has been accessed once, and the counting can start.

        If it is hot enough, the isHotKey method will be called many times.

        

1.2. Client access key

        The client is mainly the JdHotKeyStore class as the exposure point in the dependency package. The most important one is the isHotKey method, which is counted and saved locally first, and will not be sent to the server immediately.

public static boolean isHotKey(String key) {
    try {
        if (!inRule(key)) {
            return false;
        }
        boolean isHot = isHot(key);
        if (!isHot) {
            HotKeyPusher.push(key, null);
        } else {
            ValueModel valueModel = getValueSimple(key);
            //判断是否过期时间小于1秒，小于1秒的话也发送
            if (isNearExpire(valueModel)) {
                HotKeyPusher.push(key, null);
            }
        }

        //统计计数
        KeyHandlerFactory.getCounter().collect(new KeyHotModel(key, isHot));
        return isHot;
    } catch (Exception e) {
        return false;
    }

}

Basically save 0.5 seconds and send it to the server

public static void push(String key, KeyType keyType, int count, boolean remove) {
if (count <= 0) {
count = 1;
}
if (keyType == null) {
keyType = KeyType.REDIS_KEY;
}
if (key == null) {
return;
}

LongAdder adderCnt = new LongAdder();
adderCnt.add(count);

HotKeyModel hotKeyModel = new HotKeyModel();
hotKeyModel.setAppName(Context.APP_NAME);
hotKeyModel.setKeyType(keyType);
hotKeyModel.setCount(adderCnt);
hotKeyModel.setRemove(remove);
hotKeyModel.setKey(key);


if (remove) {
//如果是删除key，就直接发到etcd去，不用做聚合。但是有点问题现在，这个删除只能删手工添加的key，不能删worker探测出来的
//因为各个client都在监听手工添加的那个path，没监听自动探测的path。所以如果手工的那个path下，没有该key，那么是删除不了的。
//删不了，就达不到集群监听删除事件的效果，怎么办呢？可以通过新增的方式，新增一个热key，然后删除它
EtcdConfigFactory.configCenter().putAndGrant(HotKeyPathTool.keyPath(hotKeyModel), Constant.DEFAULT_DELETE_VALUE, 1);
EtcdConfigFactory.configCenter().delete(HotKeyPathTool.keyPath(hotKeyModel));
//也删worker探测的目录
EtcdConfigFactory.configCenter().delete(HotKeyPathTool.keyRecordPath(hotKeyModel));
} else {
//如果key是规则内的要被探测的key，就积累等待传送
if (KeyRuleHolder.isKeyInRule(key)) {
//积攒起来，等待每半秒发送一次
KeyHandlerFactory.getCollector().collect(hotKeyModel);
}
}
}

        Here, it is stored locally in ConcurrentHashMap. The more ingenious idea is to use two maps to prevent data from being written into it when the map fetches data, and change the atomic class every time the data is fetched.

public void collect(HotKeyModel hotKeyModel) {
    String key = hotKeyModel.getKey();
    if (StrUtil.isEmpty(key)) {
        return;
    }
    if (atomicLong.get() % 2 == 0) {
        //不存在时返回null并将key-value放入，已有相同key时，返回该key对应的value，并且不覆盖
        HotKeyModel model = map0.putIfAbsent(key, hotKeyModel);
        if (model != null) {
            model.add(hotKeyModel.getCount());
        }
    } else {
        HotKeyModel model = map1.putIfAbsent(key, hotKeyModel);
        if (model != null) {
            model.add(hotKeyModel.getCount());
        }
    }

}

public List<HotKeyModel> lockAndGetResult() {
    //自增后，对应的map就会停止被写入，等待被读取
    atomicLong.addAndGet(1);

    List<HotKeyModel> list;
    if (atomicLong.get() % 2 == 0) {
        list = get(map1);
        map1.clear();
    } else {
        list = get(map0);
        map0.clear();
    }
    return list;
}

When the service starts, it starts a scheduled task, and after taking out the data from the map, it will be sent out through netty

public static void startPusher(Long period) {
    if (period == null || period <= 0) {
        period = 500L;
    }
    @SuppressWarnings("PMD.ThreadPoolCreationRule")
    ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(new NamedThreadFactory("hotkey-pusher-service-executor", true));
    scheduledExecutorService.scheduleAtFixedRate(() -> {
        IKeyCollector<HotKeyModel, HotKeyModel> collectHK = KeyHandlerFactory.getCollector();
        List<HotKeyModel> hotKeyModels = collectHK.lockAndGetResult();
        if(CollectionUtil.isNotEmpty(hotKeyModels)){
            KeyHandlerFactory.getPusher().send(Context.APP_NAME, hotKeyModels);
            collectHK.finishOnce();
        }

    },0, period, TimeUnit.MILLISECONDS);
}

public void send(String appName, List<HotKeyModel> list) {
    //积攒了半秒的key集合，按照hash分发到不同的worker
    long now = System.currentTimeMillis();

    Map<Channel, List<HotKeyModel>> map = new HashMap<>();
    for(HotKeyModel model : list) {
        model.setCreateTime(now);
        Channel channel = WorkerInfoHolder.chooseChannel(model.getKey());
        if (channel == null) {
            continue;
        }

        List<HotKeyModel> newList = map.computeIfAbsent(channel, k -> new ArrayList<>());
        newList.add(model);
    }

    for (Channel channel : map.keySet()) {
        try {
            List<HotKeyModel> batch = map.get(channel);
            HotKeyMsg hotKeyMsg = new HotKeyMsg(MessageType.REQUEST_NEW_KEY, Context.APP_NAME);
            hotKeyMsg.setHotKeyModels(batch);
            channel.writeAndFlush(hotKeyMsg).sync();
        } catch (Exception e) {
            try {
                InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress();
                JdLogger.error(getClass(),"flush error " + insocket.getAddress().getHostAddress());
            } catch (Exception ex) {
                JdLogger.error(getClass(),"flush error");
            }

        }
    }

}

1.3. Server Computing

        The server listens to the netty event, and if it is a new key access, it enters the newKey method of KeyListener

public void newKey(HotKeyModel hotKeyModel, KeyEventOriginal original) {
//cache里的key
String key = buildKey(hotKeyModel);
//判断是不是刚热不久
Object o = hotCache.getIfPresent(key);
if (o != null) {
return;
}

SlidingWindow slidingWindow = checkWindow(hotKeyModel, key);
//看看hot没
boolean hot = slidingWindow.addCount(hotKeyModel.getCount());

if (!hot) {
//如果没hot，重新put，cache会自动刷新过期时间
CaffeineCacheHolder.getCache(hotKeyModel.getAppName()).put(key, slidingWindow);
} else {
hotCache.put(key, 1);

//删掉该key
CaffeineCacheHolder.getCache(hotKeyModel.getAppName()).invalidate(key);

//开启推送
hotKeyModel.setCreateTime(SystemClock.now());

//当开关打开时，打印日志。大促时关闭日志，就不打印了
if (EtcdStarter.LOGGER_ON) {
logger.info(NEW_KEY_EVENT + hotKeyModel.getKey());
}

//分别推送到各client和etcd
for (IPusher pusher : iPushers) {
pusher.push(hotKeyModel);
}

}

}

        The main thing is to open a sliding window SlidingWindow. The key is the setting rules corresponding to the two key prefixes. One is the interval and the other is the quantity, such as the machine failure key, 2s5 visits, and then the addCount method of SlidingWindow determines whether it has become hot or not

        . key

public synchronized boolean addCount(long count) {
        //当前自己所在的位置，是哪个小时间窗
        int index = locationIndex();
//        System.out.println("index:" + index);
        //然后清空自己前面windowSize到2*windowSize之间的数据格的数据
        //譬如1秒分4个窗口，那么数组共计8个窗口
        //当前index为5时，就清空6、7、8、1。然后把2、3、4、5的加起来就是该窗口内的总和
        clearFromIndex(index);

        int sum = 0;
        // 在当前时间片里继续+1
        sum += timeSlices[index].addAndGet(count);
        //加上前面几个时间片
        for (int i = 1; i < windowSize; i++) {
            sum += timeSlices[(index - i + timeSliceSize) % timeSliceSize].get();
        }

        lastAddTimestamp = SystemClock.now();

        return sum >= threshold;
    }

1.4. Hot key sending client

        After the hot key is calculated, it is not sent to the client immediately. It will be put in the queue first, and then pushed every 10ms. In fact, for a company with a small size, this is actually unnecessary

public void batchPushToClient() {
    AsyncPool.asyncDo(() -> {
        while (true) {
            try {
                List<HotKeyModel> tempModels = new ArrayList<>();
                //每10ms推送一次
                Queues.drain(hotKeyStoreQueue, tempModels, 10, 10, TimeUnit.MILLISECONDS);
                if (CollectionUtil.isEmpty(tempModels)) {
                    continue;
                }

                Map<String, List<HotKeyModel>> allAppHotKeyModels = new HashMap<>();

                //拆分出每个app的热key集合，按app分堆
                for (HotKeyModel hotKeyModel : tempModels) {
                    List<HotKeyModel> oneAppModels = allAppHotKeyModels.computeIfAbsent(hotKeyModel.getAppName(), (key) -> new ArrayList<>());
                    oneAppModels.add(hotKeyModel);
                }

                //遍历所有app，进行推送
                for (AppInfo appInfo : ClientInfoHolder.apps) {
                    List<HotKeyModel> list = allAppHotKeyModels.get(appInfo.getAppName());
                    if (CollectionUtil.isEmpty(list)) {
                        continue;
                    }

                    HotKeyMsg hotKeyMsg = new HotKeyMsg(MessageType.RESPONSE_NEW_KEY);
                    hotKeyMsg.setHotKeyModels(list);

                    //整个app全部发送
                    appInfo.groupPush(hotKeyMsg);
                }

                allAppHotKeyModels = null;

            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    });
}

1.5. The client caches hot keys

        Then it returns to the client. The client receives the hot key event and writes it to the local cache. Caffeine, the king of cache performance, is used here. If it is a fixed scene, it can be rewritten. When adding the cache, check the value according to the key. Then set it, so that there is no need to set the value when the client judges that it is a hot key and has no value.

public void newKey(HotKeyModel hotKeyModel) {
    long now = System.currentTimeMillis();
    //如果key到达时已经过去1秒了，记录一下。手工删除key时，没有CreateTime
    if (hotKeyModel.getCreateTime() != 0 && Math.abs(now - hotKeyModel.getCreateTime()) > 1000) {
        JdLogger.warn(getClass(), "the key comes too late : " + hotKeyModel.getKey() + " now " +
                +now + " keyCreateAt " + hotKeyModel.getCreateTime());
    }
    if (hotKeyModel.isRemove()) {
        //如果是删除事件，就直接删除
        deleteKey(hotKeyModel.getKey());
        return;
    }
    //已经是热key了，又推过来同样的热key，做个日志记录，并刷新一下
    if (JdHotKeyStore.isHot(hotKeyModel.getKey())) {
        JdLogger.warn(getClass(), "receive repeat hot key ：" + hotKeyModel.getKey() + " at " + now);
    }
    addKey(hotKeyModel.getKey());
}
private void addKey(String key) {
    ValueModel valueModel = ValueModel.defaultValue(key);
    if (valueModel == null) {
        //不符合任何规则
        deleteKey(key);
        return;
    }
    //如果原来该key已经存在了，那么value就被重置，过期时间也会被重置。如果原来不存在，就新增的热key
    JdHotKeyStore.setValueDirectly(key, valueModel);
}

2. Etcd storage

        As a persistent storage, Etcd mainly uses hotkeys to prevent the loss of hotkeys caused by service single point failure or release, and also needs to store hotkey rules. Each node of the server will also put its own ip information in etcd, so that the client can get the information of the server for netty push

2.1. The client updates the hot key

        Update the local cache by monitoring the data changes of etcd under the corresponding path

private void startWatchHotKey() {
    ExecutorService executorService = Executors.newSingleThreadExecutor();
    executorService.submit(() -> {
        JdLogger.info(getClass(), "--- begin watch hotKey change ----");
        IConfigCenter configCenter = EtcdConfigFactory.configCenter();
        try {
            KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyPath + Context.APP_NAME);
            //如果有新事件，即新key产生或删除
            while (watchIterator.hasNext()) {
                WatchUpdate watchUpdate = watchIterator.next();

                List<Event> eventList = watchUpdate.getEvents();
                KeyValue keyValue = eventList.get(0).getKv();
                Event.EventType eventType = eventList.get(0).getType();
                try {
                    String key = keyValue.getKey().toStringUtf8().replace(ConfigConstant.hotKeyPath + Context.APP_NAME + "/", "");

                    //如果是删除key，就立刻删除
                    if (Event.EventType.DELETE == eventType) {
                        HotKeyModel model = new HotKeyModel();
                        model.setRemove(true);
                        model.setKey(key);
                        EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model));
                    } else {
                        HotKeyModel model = new HotKeyModel();
                        model.setRemove(false);
                        String value = keyValue.getValue().toStringUtf8();
                        //新增热key
                        JdLogger.info(getClass(), "etcd receive new key : " + key + " --value:" + value);
                        //如果这是一个删除指令，就什么也不干
                        if (Constant.DEFAULT_DELETE_VALUE.equals(value)) {
                            continue;
                        }

                        //手工创建的value是时间戳
                        model.setCreateTime(Long.valueOf(keyValue.getValue().toStringUtf8()));

                        model.setKey(key);
                        EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model));
                    }
                } catch (Exception e) {
                    JdLogger.error(getClass(), "new key err ：" + keyValue);
                }

            }
        } catch (Exception e) {
            JdLogger.error(getClass(), "watch err");
        }
    });

}

2.2. Client update rules

private void startWatchRule() {
    ExecutorService executorService = Executors.newSingleThreadExecutor();
    executorService.submit(() -> {
        JdLogger.info(getClass(), "--- begin watch rule change ----");
        try {
            IConfigCenter configCenter = EtcdConfigFactory.configCenter();
            KvClient.WatchIterator watchIterator = configCenter.watch(ConfigConstant.rulePath + Context.APP_NAME);
            //如果有新事件，即rule的变更，就重新拉取所有的信息
            while (watchIterator.hasNext()) {
                //这句必须写，next会让他卡住，除非真的有新rule变更
                WatchUpdate watchUpdate = watchIterator.next();
                List<Event> eventList = watchUpdate.getEvents();
                JdLogger.info(getClass(), "rules info changed. begin to fetch new infos. rule change is " + eventList);

                //全量拉取rule信息
                fetchRuleFromEtcd();
            }
        } catch (Exception e) {
            JdLogger.error(getClass(), "watch err");
        }


    });
}

2.3. The server checks that it is in etcd

public void makeSureSelfOn() {
    //开启上传worker信息
    ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
    scheduledExecutorService.scheduleAtFixedRate(() -> {

        try {
            if (canUpload) {
                uploadSelfInfo();
            }
        } catch (Exception e) {
            //do nothing
        }

    }, 0, 5, TimeUnit.SECONDS);
}
private void uploadSelfInfo() {
    configCenter.putAndGrant(buildKey(), buildValue(), 8);
}

2.4. Check the address displayed on the front end, because the front end of hotkey is made with jsp

@Scheduled(fixedRate = 30000)
public void fetchDashboardIp() {
    try {
        //获取DashboardIp
        List<KeyValue> keyValues = configCenter.getPrefix(ConfigConstant.dashboardPath);

        //是空，给个警告
        if (CollectionUtil.isEmpty(keyValues)) {
            logger.warn("very important warn !!! Dashboard ip is null!!!");
            return;
        }

        String dashboardIp = keyValues.get(0).getValue().toStringUtf8();
        NettyClient.getInstance().connect(dashboardIp);

    } catch (Exception e) {
        e.printStackTrace();
    }
}

3. Front-end display and rule configuration

3.1. The server sends the dashboard

        When the hot key is pushed, in addition to pushing to the client, it will also be pushed to the dashboard

@PostConstruct
public void uploadToDashboard() {
    AsyncPool.asyncDo(() -> {
        while (true) {
            try {
                //要么key达到1千个，要么达到1秒，就汇总上报给etcd一次
                List<HotKeyModel> tempModels = new ArrayList<>();
                Queues.drain(hotKeyStoreQueue, tempModels, 1000, 1, TimeUnit.SECONDS);
                if (CollectionUtil.isEmpty(tempModels)) {
                    continue;
                }

                //将热key推到dashboard
                DashboardHolder.flushToDashboard(FastJsonUtils.convertObjectToJSON(tempModels));
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    });
}

3.2. The dashboard receives the hot key

        After receiving the read event through netty, store it in the queue

protected void channelRead0(ChannelHandlerContext ctx, String message) {
    if (StringUtils.isEmpty(message)) {
        return;
    }
    try {
        HotKeyMsg msg = FastJsonUtils.toBean(message, HotKeyMsg.class);
        if (MessageType.PING == msg.getMessageType()) {
            String hotMsg = FastJsonUtils.convertObjectToJSON(new HotKeyMsg(MessageType.PONG, PONG));
            FlushUtil.flush(ctx, MsgBuilder.buildByteBuf(hotMsg));
        } else if (MessageType.REQUEST_HOT_KEY == msg.getMessageType()) {
            List<HotKeyModel> list = FastJsonUtils.toList(msg.getBody(), HotKeyModel.class);
            for (HotKeyModel hotKeyModel : list) {
                HotKeyReceiver.push(hotKeyModel);
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
    }

}

        Then the data in the queue will be continuously consumed, and a copy of the data will be placed in the blocking queue and the local cache respectively.

public void dealHotKey() {
    while (true) {
        try {
            HotKeyModel model = HotKeyReceiver.take();
            //将该key放入实时热key本地缓存中
            if (model != null) {
                //将key放到队列里，供入库时分批调用
                putRecord(model.getAppName(), model.getKey(), model.getCreateTime());
                //获取发来的这个热key，存入本地caffeine，设置过期时间
                HotKeyReceiver.writeToLocalCaffeine(model);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

        Another asynchronous thread will continue to insert data into mysql

public void insertRecords() {
        while (true) {
            try {
                List<IRecord> records = new ArrayList<>();
                Queues.drain(queue, records, 1000, 1, TimeUnit.SECONDS);
                if (CollectionUtil.isEmpty(records)) {
                    continue;
                }
                List<KeyRecord> keyRecordList = new ArrayList<>(records.size());
                for (IRecord iRecord : records) {
                    KeyRecord keyRecord = handHotKey(iRecord);
                    if (keyRecord != null) {
                        keyRecordList.add(keyRecord);
                    }
                }
                if(CollectionUtil.isEmpty(keyRecordList)){
                    continue;
                }
                keyRecordMapper.batchInsert(keyRecordList);

            } catch (Exception e) {
                log.error("batch insert error:{}", e.getMessage(), e);
//                e.printStackTrace();
            }
        }

    }

3.3. Hot key rules

        The entry is in RuleController

@PostMapping("/save")
@ResponseBody
public Result save(Rules rules){
   checkApp(rules.getApp());
   checkRule(rules.getRules());
   rules.setUpdateUser(userName());
   int b = ruleService.save(rules);
   return b == 0 ? Result.fail():Result.success();
}

        When saving the rules, a copy will be inserted into etcd first, so that both the client and the server can obtain the latest rules by monitoring etcd

public int save(Rules rules) {
    String app = rules.getApp();

    KeyValue kv = configCenter.getKv(ConfigConstant.rulePath + app);
    String from = null;
    if (kv != null) {
        from = kv.getValue().toStringUtf8();
    }
    String to = JSON.toJSONString(rules);
    configCenter.put(ConfigConstant.rulePath + app, rules.getRules());

    logMapper.insertSelective(new ChangeLog(app, 1, from, to, rules.getUpdateUser(), app, SystemClock.nowDate()));
    return 1;
}

Four. Summary

        Hotkey has done a lot of detailed development for hotspot detection. If there are business scenarios that need to be changed, secondary development can be done, and bloggers can help discuss it. Here, I would like to thank JD Wu Weifeng, the creator of hotkey, again.