Detailed explanation of Kafka and detailed cases of integrating SpringBoot

Kafka is a distributed streaming media platform owned by Apache. Kafka is a high-throughput, durable, distributed publish-subscribe message queue system. It was originally released by LinkedIn (LinkedIn), written in Scala language, and open sourced in December 2010, becoming a top-level sub-project of Apache. It is mainly used to process all action flow data in consumer scale websites. Actions (data generated by web browsing, searches, and other user actions).

Classification of messaging systems

Peer-to-Peer ( Queue )

Referred to as PTP queue mode, it can also be understood as point-to-point. For example, if I send a single email to Xiao Xu, the email will be saved in the server's cloud after I send it. When Xiao Xu opens the email client and successfully connects to the cloud server, he can automatically receive the email or manually receive the email locally. , when the emails in the server cloud are consumed by Xiao Xu, they will no longer be stored in the cloud (this depends on the configuration of the email server).

Peer-to-Peer mode works:

The message producer Producer1produces the message Queue, then Consumer1takes it from the Queue and consumes the message.
After the message is consumed, Queuethe message will no longer be stored, and it Consumeris impossible to consume messages that have been consumed by other Consumers.
QueueMultiple messages are supported Producer, but for a message, only one Consumercan be consumed, and other Consumers cannot consume it again.
But Consumerwhen it does not exist, the message is Queuesaved until Consumerit is consumed.

Publish/Subscribe modePublish /Subscribe (Topic)

Referred to as the publish/subscribe model. For example, I have 300,000 fans on Weibo and I updated a Weibo today. Then all 300,000 fans can receive my Weibo updates and everyone can consume my messages.

Publish/Subscribe mode works:

The message publisher Publisherpublishes the message to the topic Topic, and multiple message consumers Subscriberconsume the message at the same time.
Unlike the PTP method, Topicmessages published will be consumed by all subscribers.
When a publisher publishes a message, no error message will be reported regardless of whether there are subscribers.
There must be a message publisher first, and then a message subscriber.

Kafka features

High throughput : can meet the production and consumption of millions of messages per second.

Persistence : There is a complete message storage mechanism to ensure data is efficient, secure and durable.

Distributed : Based on distributed expansion; Kafka data will be copied to several servers. When a certain server fails, producers and consumers will switch to other Kafkas.

Scalability : kafka cluster supports hot expansion

Introduction to kafka architecture

We can see in the picture above that Broker and Consumer use ZooKeeper, but Producer does not use ZooKeeper, because starting from Kafka version 0.8, Producer does not need to obtain the cluster status based on ZooKeeper, but specifies multiple Brokers in the configuration. The node sends a message and establishes a connection with the specified Broker to obtain the status information of the cluster from the Broker. This allows the Producer to know how many Brokers are alive in the cluster and how many Partitions there are in the Topic on each Broker. , Prodocuer will store these meta-information into Producer's memory. If Producer times out when sending information to a Broker node in the cluster, Producer will actively refresh the meta-information in the memory to obtain the latest status of the current Broker cluster, and then send the information to the currently available Broker. Of course, Producer You can also specify in the configuration to periodically refresh the Broker's metainformation to update it to the memory. Note: Only Broker and ZooKeeper are services, while Producer and Consumer are just Kafka’s SDK.

Introduction to related terms

noun	explain
Producer	The producer of messages and data is mainly responsible for producing `Push`messages to the Topic of the designated Broker.
Broker	Kafka nodes are called Brokers. Brokers are mainly responsible for creating Topics, storing messages published by Producers, recording the message processing process, and currently saving messages to memory and then persisting them to disk.
Topic	Messages of the same Topic can be distributed on one or more Brokers. A Topic contains one or more Partition partitions, and data is stored in multiple Partitions.
replication-factor	Replication factor; this term never appears in the above figure. This option will be specified when we create a Topic in the next chapter, which means whether a copy is required to create the current Topic. If this value is set to 1 when creating a Topic, It means that there is only one copy of the entire Topic in Kafka. It is recommended that the number of replication factors be consistent with the number of Broker nodes.
Partition	Partition; here it is called the physical grouping of Topic. A Topic is divided into one or more Partitions in the Broker. It can also be said that each Topic contains one or more Partitions (usually Kafka nodes. Points CPU The total number of cores) partition can be specified when creating the Topic. Partitions are the actual units where data is stored.
Consumer	Consumers of messages and data are mainly responsible for actively pulling messages from subscribed topics and consuming them. Why can't the Consumer push data through the Broker like the Producer? Because the Broker does not know how much the Consumer can consume, if the amount of push message data is too much, it will cause message blocking. If the Consumer actively pulls the data, the Consumer can pull the message data according to its own processing situation, and retrieve the message again after consuming it. . This will not cause the data that the Consumer itself has obtained to become blocked.
ZooKeeper	ZooKeeper is responsible for maintaining the status of the entire Kafka cluster, storing the information and status of each Kafka node, achieving high availability of the Kafka cluster, and coordinating the work content of Kafka.

Basic principles of kafka

We call the publishing of the message (publish) as the producer, the subscription of the message (subscribe) as the consumer, and the intermediate storage array as the broker (agent), so that we can roughly depict such a scene:

The producer produces the data and hands it to the broker for storage. When the consumer needs to consume the data, he takes the data from the broker and completes a series of data processing operations.

Multiple brokers work together, and producer and consumer deployments are frequently called in various business logics. The three coordinate requests and forwarding through zookeeper management. In this way, a high-performance distributed message publishing and subscription system is completed.

There is a detail to note in the figure. The process from producer to broker is push, that is, when data is available, it is pushed to broker. The process from consumer to broker is pull, which means the consumer actively pulls data.

kafka application scenarios

1. Log collection

A company can use Kafka to collect logs of various services, and use Kafka to open them to various consumers in the form of unified interface services, such as Hadoop, Hbase, Solr, etc.

2.Message system

Decoupling producers and consumers, caching messages, etc.

3. User activity tracking

Kafka is often used to record various activities of web users or app users, such as browsing the web, searching, clicking and other activities. These activity information are published by various servers to Kafka topics, and then subscribers do real-time by subscribing to these topics. Monitor and analyze, or load it into Hadoop or data warehouse for offline analysis and mining.

4. Operational indicators

Kafka is also often used to record operational monitoring data. Including collecting data from various distributed applications and producing centralized feedback for various operations, such as alarms and reports.

5. Streaming

Such as spark streaming and storm

kafka core API

Producer API : The Producer API allows applications to publish a set of records to one or more Kafka Topics.
Consumer API : The Consumer API allows applications to subscribe to one or more Topics and process the stream of records delivered to them.
Streams API : The Streams API allows applications to act as stream processors, consuming input streams from one or more Topics and generating output streams into one or more output topics, thereby effectively converting input streams into output streams.
Connector API : The Connector API allows building and running reusable producers or consumers that connect Kafka Topics to existing applications or data systems. For example: a connector to a relational database might capture every change to a table.

Detailed explanation of kafka components

Topics and Logs

Topics and logs are officially called yes Topic and log. Topic is the name of the category or feed to which the record is published. Kafka's Topic is always multi-user; that is, a Topic can have zero, one or more consumers subscribing to the data written to it. Each Topic Kafka cluster has a Partition partition log, as shown in the following figure:

Each Partition partition is an ordered sequence of records (immutable). If there is a new log, it will be added to the end in a sequential structure. Each record in the partition is assigned an ID number in sequence, which is called a partial The shift amount is unique in the entire Partition. As shown in the figure above, there are Partition, Partition1, and Partition2. The order of log writing is from Old to New, and the ID number is from 0-12, etc.

Message records published by the Kafka cluster will be persisted to the hard disk. Regardless of whether the message is consumed or not, the published records will be retained by Kafka on the hard disk. We can set the retention period. For example, if the retention policy is set to two days, the message will be available within two days of publishing the record, and then discarded by Kafka to free up space. Kafka's performance is excellent in terms of data size and can be retained for a long time Data is not a problem.

In fact, the only metadata retained on a per-consumer basis is the consumer's offset or position in the log. This offset is controlled by the consumer: the consumer will usually advance the offset linearly as it reads records, but in fact, since the position of the consumer is controlled by the consumer, it can move any way it likes. Consumption records are performed sequentially. For example, a consumer can reset the previous offset to process previous data, or start consuming directly from the latest offset. The combination of these features means that Kafka consumers are very trivial and they can be as casual as they want without having much impact on the cluster or other consumers. For example: you can use the command tool to "tail" the content of any Topic without changing the content used by any existing users.

Partitioning in the log serves several purposes. First, they allow logs to be larger than what fits on a single server, each individual partition must fit on the server hosting it, but a topic may have many partitions so it can handle any amount of data, and second, they serve as parallel Units - more on that in a bit.

Distribution

Log Partition partitions are distributed across servers in the Kafka cluster, and each server processes data and requests shared partitions. To achieve fault tolerance, each partition is replicated to multiple servers in a configurable Kafka cluster.

Noun introduction:

leader:leader

followers: follower

Each Partition partition has a leader(leader) server, which is each Partition partition. If our Partition1 partition is copied to three servers, the second one is the leader of this Partition partition, and the other two servers will Become a (follower) of this Partition followers. leaderThe (leader) of the Partition shard handles all read and write requests for the Partition partition, while followerthe (follower) passively replicates leaderthe changes made by the (leader). If the leader of the Partition shard fails, etc., followerOne of the two (followers) servers will automatically become the new leaderleader. Each server acts as leadera leader for some partitions and followera follower for some partitions, so the load within the cluster is very balanced.

leaderNote: The sums mentioned above followerare only the leaders and followers of each Partition partition, not the master node and backup node of the entire cluster that we learned before. I hope you will not be confused.

Geo-Replication (regional replication)

Kafka Mirrormaker provides geographical replication support for clusters. Using MirrorMaker, you can replicate data across multiple computer rooms or clouds. You can use it for backup and recovery in an active/passive solution; in an active solution, you can keep data closer to users, or Supports data location requirements.

Producers(producers)

Producers publish data to the Topic of their choice, and the producer is responsible for selecting which partition in the Topic the records are assigned to. This can be done in a round-robin fashion, just for load balancing, or it can be done based on some semantic partitioning function (such as based on a certain key in the record).

Consumers

Introduction to nouns

Consumers: consumers

Consumers Group: Consumer Group

Consumers Group name: Consumer group name

ConsumersUse Consumers Group namethe tag yourself, and each record published to the Topic is delivered to Consumers Groupa Consumers instance in each subscription. The Consumers instances can be in separate processes or on different machines. If all Consumers instances have the same record Consumers Group, Will effectively load balance across Consumers.

If all Consumers instances are in different processes Consumers Group, each record will be broadcast to all Consumers processes.

Two Kafka Clusters host four Partitions (partitions), from P0-P3, including two Consumers Groupand Consumer Group Atwo Consumers instances, and B has four Consumers instances Consumer Group B. Consumners Group AThat is, consumer group A has two consumers, and group B has four consumers. Then, more commonly, we find that there are a small number of Topics Consumers Group, each consumer corresponds to a user group, and each group is composed of many consumer instances for scalability and fault tolerance. This is nothing more than publish/subscribe semantics, where A subscriber is a group of consumers, not a single process.

The way consumers are implemented in Kfaka is by dividing the partition in the log on the consumer instance so that each instance has the "same share" of the allocation at any point in time. The process of maintaining the successful qualifications of the consumer group is dynamically performed by Kafka The protocol is implemented. If a new consumer instance joins the consumer group, the new consumer instance will take over some partitions from other members of the group; if the consumer instance fails, its partitions will be distributed to the remaining consumer instances.

Kafka only provides the total order of records within a partition, not records between different partitions in a Topic. For most applications, the ability to sort by partition and partition data by key is sufficient, but if you need to record the total order, this can be achieved using a topic with only one partition, although this means that each consumer group only has A consumer process.

Consumer Group

We talked about the classification of message systems at the beginning: PTP mode and publish/subscribe mode. We also mentioned that our Kafka adopts the publish-subscribe mode, that is, a message producer generates messages into a Topic, and all consumers can consume them. This message adopts the asynchronous model; while PTP means that the message produced by a message producer cannot be sent to the Queue and can only be consumed by a message consumer. Using the synchronous model, in fact, the publish-subscribe model can also implement the PTP model. Multiple consumers join a consumer group. For example, there are three consumer groups, and each group has 3 message consumer instances. That is, there are a total of 9 consumer instances. If there is a message to be received in the Topic, The three consumer groups will all receive messages, but each one receives a message, and then the consumer group sends the message to a consumer instance in the group, not all consumer instances. In the end, only Three consumer instances got this message. Of course, we can also add these consumers to only one consumer group, so that only one consumer can get the message.

Guarantees

The following guarantees are provided in high-level Kafka:

Messages sent by a producer to a specific Topic partition will be appended in the order they were sent. In other words, if a Producer sends M1 and M2, generally speaking in order, M1 must be sent first, followed by M2. If this is the case, if the number of M1 is 1 and the number of M2 is 2, Then there will be M1 in the log, followed by M2.
Consumer instances view records in the order they are stored in the log.
For a topic with a replication factor N, Kafka can tolerate up to N-1 server failures without losing any records submitted to the log.

kafka messaging process

When the Producer publishes a message to a Partition, it first finds the Leader of the Partition through Zookeeper, and then no matter what the Replication Factor of the Topic is (that is, how many Replicas the Partition has), the Producer only sends the message to the Leader of the Partition. . The Leader will write the message to its local Log. Each Follower pulls data from the Leader. In this way, the order of data stored by Follower is consistent with that of Leader. After receiving the message and writing it to its Log, the Follower sends an ACK to the Leader. Once the Leader receives ACKs from all Replicas in the ISR, the message is considered committed, and the Leader will increase the HW (offset) and send ACK to the Producer.

In order to improve performance, each Follower sends an ACK to the Leader immediately after receiving the data instead of waiting for the data to be written to the Log. Therefore, for messages that have been committed, Kafka can only guarantee that they are stored in the memory of multiple Replicas, but cannot guarantee that they will be persisted to disk. It cannot fully guarantee that the message will be consumed by the Consumer after an exception occurs. . But considering that this scenario is very rare, it can be considered that this method strikes a better balance between performance and data persistence. In future versions, Kafka will consider providing higher durability. The Consumer also reads messages from the Leader. Only messages that have been committed (messages with an offset lower than the HW) will be exposed to the Consumer.

The specific steps are summarized as follows:

The producer first finds the leader of the partition from the "/brokers/.../state" node of zookeeper.
The producer sends the message to the leader
The leader writes the message to the local log
Followers pull messages from the leader, write them to the local log, and then send ACK to the leader.
After the leader receives ACKs from all replicas in the ISR, it increases HW (high watermark, the offset of the last commit) and sends ACK to the producer.

kafka election mechanism

Controller (Broker) election Leader mechanism

The so-called controller is a Borker. In a Kafka cluster, there are multiple broker nodes, but a leader needs to be elected between them, and other brokers act as followers. The first broker to start in the cluster will make itself the controller by creating a temporary node/controller in zookeeper. Other brokers will also create temporary nodes in zookeeper when they start, but they find that the node already exists, so they will receive a Exception, realizing that the controller already exists, then a watch object will be created in zookeeper so that they can receive notification of controller changes.

Then if the controller disconnects from ZooKeeper or exits abnormally due to network reasons, other brokers will receive notifications of controller changes through watch and will try to create temporary nodes/controllers. If one broker is successfully created, then other brokers will You will receive a creation exception notification, which means that there is already a controller in the cluster, and other brokers only need to create watch objects.
If a broker in the cluster exits abnormally, the controller will check whether the broker has a replica leader of the partition . If so, the partition needs a new leader. At this time, the controller will traverse other replicas and decide Which one becomes the new leader and updates the partition's ISR set at the same time.
If a broker joins the cluster, the controller will use the Broker ID to determine whether the newly added broker contains a copy of the existing partition. If so, it will synchronize the data from the partition copy.
Every time a controller is elected in the cluster, a controller epoch will be created through zookeeper. Each election will create a larger epoch containing the latest information. If a broker receives data older than this epoch, it will ignore them. Kafka This epoch is also used to prevent "split brain" in the cluster .

Partition copy leader election mechanism

In the Kafka cluster, there will be multiple topics. Each topic is divided into multiple partitions. In order to prevent data from being lost, each partition has multiple copies. In the entire cluster, there are a total of three Copy role:

Leader copy : That is, the leader master copy. Each partition has a leader copy. In order to ensure data consistency, all producer and consumer requests will be processed through this copy.
Follower copy : All copies except the leader copy are follower copies. The follower copy does not process any requests from the client and is only responsible for synchronizing data from the leader copy to ensure consistency with the leader. If the leader replica crashes, a leader will be elected from it.
Priority replica : The priority leader specified when creating a partition. If not specified, the first copy of the partition.

By default, if no request is sent between the follower and the leader for more than 10 seconds, or the request data is not received, the follower will be considered an "out of sync copy". The continuously requested copy is the "synchronized copy". When the leader fails, only the "synchronized copy" can be elected as the leader . The request timeout can be configured through the parameter replica.lag.time.max.ms parameter.

We hope that the leaders of each partition can be distributed to different brokers to achieve load balancing as much as possible, so there will be a priority leader. If we set the parameter auto.leader.rebalance.enable to true, then it will check whether the priority leader is the real leader. If not, an election will be triggered to make the priority leader the leader.

Consumer group election mechanism

The group coordinator will elect a leader for all consumers in the consumer group. The algorithm of this election is also very simple. The first consumer to join the consumer group is the leader. If the leader consumer exits at a certain moment Consumer group, then a new leader will be randomly elected.

The role of zookeeper

Zookeeper is specifically used:

1. When the broker starts, create a temporary node znode under /brokers/ids and write the broker id into it. If the broker goes down or does not respond, the node will be deleted.

2. Each Broker will register the partition information it stores and maintains under the /broker/topics/... path.

3. Consumers will also register temporary nodes on zookeeper to maintain consumer load balancing and offset records.

4. Each consumer creates a temporary consumer_id under /consumers/[group_id]/ids, which is used to describe which consumers in the current group are alive.

5. Multiple consumers with the same group ID form a consumption group, consume a topic together, and try to balance consumption.

1) Record and maintain broker status

Zookeeper records the survival status of all brokers, and brokers will send heartbeat requests to zookeeper to report their status.

Zookeeper maintains a list of brokers that are running and belong to the cluster.

2) Controller (leader) election

There are multiple brokers in the Kafka cluster, one of which will be elected as the controller. The controller is responsible for managing the status of all partitions and replicas in the entire cluster. For example, if the leader of a certain partition fails, the controller will elect a new leader. Zookeeper is responsible for selecting controllers from multiple brokers.

3) Limit permissions

Kafka allows some clients to have different production and consumption limits. These limit configuration information is stored in zookeeper.

4) Record ISR (synchronized replica)

Kafka maintains a set called an ISR (in-sync replica, synchronized copy) on Zookeeper for each Topic. This set is a copy of some partitions. Only when these replicas are synchronized with the replicas in the Leader, Kafka will consider that the message has been submitted and feed it back to the producer of the message. If this collection increases or decreases, kafka will update the record on zookeeper . Zookeeper found that one of the members was abnormal and removed it immediately .

5) node and topic registration

Zookeeper saves the registration information of all nodes and topics, making it easy to find which topics each broker holds.

Node and topic exist in the form of temporary nodes in zookeeper. As soon as the session with zookeeper is closed, their information will be gone.

6) topic configuration

Zookeeper saves topic-related configurations, such as topic list, number of partitions for each topic, location of replicas, etc.

Specific use of kafka combined with springboot

Dependency import

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>

Configuration file

spring:
  kafka:
    bootstrap-servers: fairytail.ltd:9092
    producer:
      # 发生错误后，消息重发的次数。
      retries: 0
      #当有多个消息需要被发送到同一个分区时，生产者会把它们放在同一个批次里。该参数指定了一个批次可以使用的内存大小，按照字节数计算。
      batch-size: 16384
      # 设置生产者内存缓冲区的大小。
      buffer-memory: 33554432
      # 键的序列化方式
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      # 值的序列化方式
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      # acks=0 ： 生产者在成功写入消息之前不会等待任何来自服务器的响应。
      # acks=1 ： 只要集群的首领节点收到消息，生产者就会收到一个来自服务器成功响应。
      # acks=all ：只有当所有参与复制的节点全部收到消息时，生产者才会收到一个来自服务器的成功响应。
      acks: 1
    consumer:
      # 自动提交的时间间隔 在spring boot 2.X 版本中这里采用的是值的类型为Duration 需要符合特定的格式，如1S,1M,2H,5D
      #auto-commit-interval: 50
      # 该属性指定了消费者在读取一个没有偏移量的分区或者偏移量无效的情况下该作何处理：
      # latest（默认值）在偏移量无效的情况下，消费者将从最新的记录开始读取数据（在消费者启动之后生成的记录）
      # earliest ：在偏移量无效的情况下，消费者将从起始位置读取分区的记录
      auto-offset-reset: latest
      # 是否自动提交偏移量，默认值是true,为了避免出现重复数据和数据丢失，可以把它设置为false,然后手动提交偏移量
      enable-auto-commit: false
      # 键的反序列化方式
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      # 值的反序列化方式
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    listener:
      # 在侦听器容器中运行的线程数。
      concurrency: 8
      #listner负责ack，每调用一次，就立即commit
      ack-mode: manual_immediate
      missing-topics-fatal: false
  profiles:
    active: dev
server:
  port: 3435

Configuration class

@Configuration
public class KafkaConfig {

    @Bean
    public NewTopic batchTopic() {
        return new NewTopic("sinosun", 2, (short) 1);
    }
}

producer

@Component
public class KafkaProduce {

    @Autowired
    private KafkaTemplate<String, Object> kafkaTemplate;

    //自定义topic
    public static final String TOPIC_TEST = "sinosun";


    public static final String TOPIC_GROUP1 = "topic.group1";


    public void send(Object obj) {
        kafkaTemplate.send(TOPIC_TEST, 0,""+ System.currentTimeMillis(),obj);
    }

}

consumer

@Component
@Slf4j
public class KafkaConsumer {

    @KafkaListener(topics = KafkaProduce.TOPIC_TEST,groupId = KafkaProduce.TOPIC_GROUP1)
    public void topicTest(ConsumerRecord<?, ?> record, Acknowledgment ack, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
        Optional message = Optional.ofNullable(record.value());

        message.ifPresent((Object msg) ->{
            msg=message.orElse(null);
            log.info("Topic:" + topic + ",Message:" + msg+" partition"+record.partition()+"   class"+msg.getClass());
            String task = JSONUtil.toJsonStr(msg);
            List<Integer> integers = JSONArray.parseArray(task, Integer.class);
            integers.forEach(param->{
                ThreadPoolManager.addExecuteTask("java", param);
            });
            ack.acknowledge();
        });
    }
}

Test class

@RestController
@RequestMapping("/kafka")
@Slf4j
public class Controller {

    @Autowired
    private KafkaProduce kafkaProduce;

    @GetMapping("/test")
    public String test() throws InterruptedException {
        for (Integer i = 0; i < 100; i++) {
            List<Integer> integers = Arrays.asList(1, 2, 3, 4, 5, 6);
            String parse = JSONObject.toJSONString(integers);
            kafkaProduce.send(parse);
        }
        ThreadPoolManager.COUNT.await();
        log.info(">>>>>>.进入缓冲队列的次数[{}]", ThreadPoolManager.number);
        ThreadPoolManager.number.set(0);
        return "success";
    }

Thread Pool

/**
 * @author lijianwei
 * 线程池管理类
 */
@Slf4j
public class ThreadPoolManager {


    public static Map<String, Consumer<Object>> consumerMap = new HashMap<>();

    //最小线程数（核心线程数）
    private static int SIZE_CORE_POOL = 20;

    //最大线程数
    private static int SIZE_MAX_POOL = 20;

    //线程空闲存活时间
    private static int TIME_KEEP_ALIVE = 5000;

    //缓冲队列大小
    private static int SIZE_WORK_QUEUE = 1;

    //原子类进行拒绝策略计数
    public static AtomicInteger number = new AtomicInteger();

    //计数器，给main线程判断线程池是否执行完
    public static final CountDownLatch COUNT = new CountDownLatch(600);


    /**
     * 自定义拒绝策略，将超过线程数量的任务放入缓冲队列
     */
    private static final RejectedExecutionHandler mHandler = new RejectedExecutionHandler() {
        @SneakyThrows
        @Override
        public void rejectedExecution(Runnable task, ThreadPoolExecutor executor) {
            number.incrementAndGet();
            mThreadPool.getQueue().put(task);
            log.error("进入缓冲队列");
        }
    };


    private static ThreadPoolExecutor mThreadPool;

    /**
     * 线程池初始化
     */
    public static void init() {
        log.info(">>>>>>>>>线程池初始化");
        mThreadPool = new ThreadPoolExecutor
                (SIZE_CORE_POOL,
                 SIZE_MAX_POOL,
                 TIME_KEEP_ALIVE,
                 TimeUnit.SECONDS,
                 new ArrayBlockingQueue<>(SIZE_WORK_QUEUE),
                 Executors.defaultThreadFactory(),
                 mHandler);
    }


    /**
     * 根据不同的type进行不同的方法，利用Java的consumer类型
     * @param type
     * @param data
     */
    public static void addExecuteTask(String type, Object data) {
        try {
            mThreadPool.execute(() -> {
                try {
                    //根据类型执行不同的任务
                    consumerMap.get(type).accept(data);
                } catch (Exception e) {
                    log.error("线程池执行任务失败");
                }
            });
        } finally {
            COUNT.countDown();
        }
    }
}