Sentinel Slot extended practice - flow control fuse achieve early warning

Foreword

A few days ago the company a production environment due to the increased traffic service triggered a flow control mechanism Sentinel, and user feedback slow access, positioning is found to lead to task timed task, then release back task optimization, traffic returned to normal.

This is a perfectly normal production problems, most of the students may have experienced, experienced most of it nothing to solve the problem after, leading to possible accidents as well as recurrence, ultimately resulting in a bad user experience . So I think all the production problems need to be re-set, of course, the purpose is not to re-set accountable, but to prevent recurrence of the same mistakes next time. After that we simply analyze the problem, of course, is the operational level of oversight leads to an unreasonable task of issuing a large number of requests, and the second we simply rude flow control flow control, there is no better early warning measures, leading to affect the user we know (ie, flow control or fuse has been triggered).

That our solution? Of course, is the operational level of prevention, but this is not really the point of this article, the discussion here is not. The second is the warning that we can shortly before its trigger flow control, then an alarm to the relevant person in charge of early intervention treatment, to prevent the trigger flow control fuse. Of course, it can not be completely avoided, but better than flow control after the alarm is triggered or fuse is much better.

Because before Ali's Sentinel flow control use, so concrete realization of this paper is to use a custom slot in Sentinel, and this custom slot card slot Sentinel official document which would take over a word, and then add a demo code, I in the use of the process also encountered many pit, so the results for everyone to share.

If we are not very understanding of the Sentinel, you can go to github simply try to understand in reading this article. github address: https: //github.com/alibaba/Sentinel

If you want to be familiar with a custom slot feature suggestions's look at how the Sentinel: https: //github.com/alibaba/Sentinel/wiki/Sentinel%E5%B7%A5%E4%BD%9C%E4%B8%BB%E6 % B5% 81% E7% A8% 8B

As well as the source code of the demo for custom slot wording: https: //github.com/alibaba/Sentinel/tree/master/sentinel-demo/sentinel-demo-slot-chain-spi

Implementation

Here are the relevant Sentinel achieved under the early warning function, provided you are using the system already in use such as flow control or blown Sentinel features.

1. Custom CustomSlotChainBuilder achieve SlotChainBuilder interfaces, here mainly to our custom Slot added SlotChain this chain

import com.alibaba.csp.sentinel.slotchain.ProcessorSlotChain;
import com.alibaba.csp.sentinel.slotchain.SlotChainBuilder;
import com.alibaba.csp.sentinel.slots.DefaultSlotChainBuilder;
import com.qiaofang.tortoise.gateway.component.ApplicationContextUtil;
import com.qiaofang.tortoise.gateway.config.SentinelProperties;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;

/**
 * 自定义slot
 *
 * @author chenhao
 */
public class CustomSlotChainBuilder implements SlotChainBuilder {


    @Override
    public ProcessorSlotChain build() {
        ProcessorSlotChain chain = new DefaultSlotChainBuilder().build();
        SentinelProperties sentinelProperties = (SentinelProperties) ApplicationContextUtil.getContext().getBean("sentinelProperties");
        chain.addLast(new FlowEarlyWarningSlot(sentinelProperties));
        chain.addLast(new DegradeEarlyWarningSlot(sentinelProperties));
        return chain;
    }
}

2. Custom FlowEarlyWarningSlot, DegradeEarlyWarningSlot fluidic warning fuse slot 2

import com.alibaba.csp.sentinel.context.Context;
import com.alibaba.csp.sentinel.node.DefaultNode;
import com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot;
import com.alibaba.csp.sentinel.slotchain.ResourceWrapper;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleChecker;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleUtil;
import com.alibaba.csp.sentinel.util.AssertUtil;
import com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeanUtils;
import org.springframework.util.CollectionUtils;

import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

/**
 * 流控预警slot
 *
 * @author chenhao
 */
public class FlowEarlyWarningSlot2 extends AbstractLinkedProcessorSlot<DefaultNode> {

    /**
     * log
     */
private Logger logger = LoggerFactory.getLogger(this.getClass());

    private final FlowRuleChecker checker;

    public FlowEarlyWarningSlot2() {
        this(new FlowRuleChecker());
    }

    /**
     * Package-private for test.
     *
     * @param checker flow rule checker
     * @since 1.6.1
     */
FlowEarlyWarningSlot2(FlowRuleChecker checker) {
        AssertUtil.notNull(checker, "flow checker should not be null");
        this.checker = checker;
    }


    private List<FlowRule> getRuleProvider(String resource) {
        // Flow rule map should not be null.
        List<FlowRule> rules = FlowRuleManager.getRules();
        List<FlowRule> earlyWarningRuleList = Lists.newArrayList();
        for (FlowRule rule : rules) {
            FlowRule earlyWarningRule = new FlowRule();
            BeanUtils. CopyProperties (rule, earlyWarningRule);
             / **
              * This is equivalent to the original rule change the threshold value 80%, the effect becomes the advance warning,
              * It is recommended that the configuration is made 0.8
              * /
 earlyWarningRule.setCount (rule.getCount () * 0.8);
            earlyWarningRuleList.add(earlyWarningRule);
        }
        Map<String, List<FlowRule>> flowRules = FlowRuleUtil.buildFlowRuleMap(earlyWarningRuleList);
        return flowRules.get(resource);
    }

    /**
     * get origin rule
     *
     * @param resource
* @return
*/
private FlowRule getOriginRule(String resource) {
        List<FlowRule> originRule = FlowRuleManager.getRules().stream().filter(flowRule -> flowRule.getResource().equals(resource)).collect(Collectors.toList());
        if (CollectionUtils.isEmpty(originRule)) {
            return null;
        }
        return originRule.get(0);
    }

    @Override
    public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
            throws Throwable {
        String resource = context.getCurEntry().getResourceWrapper().getName();
        List<FlowRule> rules = getRuleProvider(resource);
        if (rules != null) {
            for (FlowRule rule : rules) {
                // Here the rules are configured to take the 80% threshold, where the threshold if it is checked, indicating that the real threshold of 80%, either send an alarm to the corresponding person in charge of the
                if (!checker.canPassCheck(rule, context, node, count, prioritized)) {
                    FlowRule originRule = getOriginRule(resource);
                    String originRuleCount = originRule == null ? "未知" : String.valueOf(originRule.getCount());
                    logger.info ( "FlowEarlyWarning: {} current service flow has exceeded index {}, close flow control configured threshold: {},", resource, rule.getCount (), originRuleCount);
                    // TODO alarm function self-realization
                    break;
                }
            }
        }
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
    }

    @Override
    public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
        fireExit(context, resourceWrapper, count, args);
    }
}

import com.alibaba.csp.sentinel.context.Context;
import com.alibaba.csp.sentinel.node.DefaultNode;
import com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot;
import com.alibaba.csp.sentinel.slotchain.ResourceWrapper;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;
import com.alibaba.csp.sentinel.util.AssertUtil;
import com.google.common.collect.Lists;
import com.qiaofang.tortoise.gateway.config.SentinelProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeanUtils;
import org.springframework.util.CollectionUtils;

import java.util.List;
import java.util.stream.Collectors;

/**
 * 熔断预警slot
 *
 * @author chenhao
 */
public class DegradeEarlyWarningSlot2 extends AbstractLinkedProcessorSlot<DefaultNode> {

    /**
     * log
     */
private Logger logger = LoggerFactory.getLogger(this.getClass());

    / **
      * is consistent with the flow control rule take the original is not the same
      * @param Resource
 * @return
 * /
 Private List <DegradeRule> getRuleProvider (String Resource) {
        // Flow rule map should not be null.
        List<DegradeRule> rules = DegradeRuleManager.getRules();
        List<DegradeRule> earlyWarningRuleList = Lists.newArrayList();
        for (DegradeRule rule : rules) {
            DegradeRule earlyWarningRule = new DegradeRule();
            BeanUtils.copyProperties(rule, earlyWarningRule);
            earlyWarningRule.setCount(rule.getCount() * 0.8);
            earlyWarningRuleList.add(earlyWarningRule);
        }
        return earlyWarningRuleList.stream().filter(rule -> resource.equals(rule.getResource())).collect(Collectors.toList());
    }

    /**
     * get origin rule
     *
     * @param resource
* @return
*/
private DegradeRule getOriginRule(String resource) {
        List<DegradeRule> originRule = DegradeRuleManager.getRules().stream().filter(rule -> rule.getResource().equals(resource)).collect(Collectors.toList());
        if (CollectionUtils.isEmpty(originRule)) {
            return null;
        }
        return originRule.get(0);
    }

    @Override
    public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
            throws Throwable {
        String resource = context.getCurEntry().getResourceWrapper().getName();
        List<DegradeRule> rules = getRuleProvider(resource);
        if (rules != null) {
            for (DegradeRule rule : rules) {
                if (!rule.passCheck(context, node, count)) {
                    DegradeRule originRule = getOriginRule(resource);
                    String originRuleCount = originRule == null ? "未知" : String.valueOf(originRule.getCount());
                    logger.info ( "DegradeEarlyWarning:} {Service current index has exceeded {} fuse, the fuse is disposed close to the threshold value: {},", resource, rule.getCount (), originRuleCount);
                    break;
                }
            }
        }
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
    }

    @Override
    public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
        fireExit(context, resourceWrapper, count, args);
    }

3. In the resources folder below META-INF.services new folder, the new file com.alibaba.csp.sentinel.slotchain.SlotChainBuilder (file name does not matter) reads as follows

# Here to write a complete package path of your CustomSlotChainBuilder
com.xxx.sentinel.CustomSlotChainBuilder

Here basically on it, the process used in or encountered a lot of the pit, simply to name a few bars

• After FlowRule direct change count property is no good, since the underlying validation rules used when the controller is FlowRule property that is private, so to get directly to regenerate the original configuration by FlowRuleUtil
• The debugging process, the value of which DefaultNode are many ways within 1s is valid, Debug from Method A to Method B values ​​may be gone, when the force look ignorant

Written in the last

    I rarely write such a technology blog, so what is the problem, not rigorous or place, we can put forward, seeking tap sprayed me ha ha ha

 

 Source: webmaster platform