Foreword:
In this article, the blog mainly introduces the principles and technologies of inter-process and intra-process communication between workers in Storm. The entire content is for understanding only, if there are any deficiencies or errors, please point out.
Communication between workers often needs to be carried out across nodes through the network . Storm uses ZeroMQ or Netty (default after 0.9) as the message framework for inter-process communication.
Worker intra-process communication: The thread communication of different workers is done using LMAX Disruptor.
Storm is not responsible for the communication between different topologeys, and needs to find a way to achieve it, such as using kafka.
1. The principle of communication between worker processes
The message passing mechanism between worker processes, the general flow of message reception and processing is shown in the following figure
1. For worker processes, in order to manage incoming and outgoing messages, each worker process has an independent receiving thread < A worker process runs a dedicated receiving thread to be responsible for moving externally sent messages to the corresponding executor In the incoming-queue of the thread> (monitoring the configured TCP port supervisor.slots.ports) corresponds to the worker receiving thread, and each worker has an independent sending thread (each element of the transfer-queue actually represents a collection of tuples ) , which is responsible for reading messages from the worker's transfer-queue and sending them to other workers over the network.
2. Each executor has its own incoming-queue and outgoing-queue.
The worker receiving thread passes the received message to the incoming-queues of the corresponding executor (one or more) through the task number;
each executor has a separate thread to process the business logic of spout/bolt and the intermediate data output by the business logic It will be stored in the outgoing-queue. When the tuple in the outgoing-queue of the executor reaches a certain threshold, the sending thread of the executor will obtain the tuple in the outgoing-queue in batches and send them to the transfer-queue.
3. Each worker process controls one or more executor threads, which can be configured by the user in the code. In fact, it is the number of concurrency we set in the code.
Summary: Each work has a corresponding receiving thread to receive messages sent from the outside through the network, and matches the corresponding excutor according to the taskId contained in the tuple, and moves the message to the incoming-queue of the corresponding excutor thread. After the executor processes the data from the incoming-queue (consume the tuple data in the incoming-queue through the bolt's excute method), it outputs the intermediate results to the outgoing-queue. When the data in the outgoing-queue reaches a certain threshold , the sending thread of the excutor will send the data in the outgoing-queue to the transfer-queue of the worker, and the sending thread of the worker will send the message read in the transfer-queue to other workers through the network.
2. Worker inter-process technology (Netty, ZeroMQ)
2.1. Netty
Netty is a NIO client-server (client-server) framework . Using Netty, you can quickly develop network applications, such as server and client protocols. Netty provides a new way to develop network applications that makes it easy to use and highly extensible. Netty's internal implementation is complex, but Netty provides an easy-to-use API to decouple business logic from network processing code. Netty is completely implemented based on NIO, so the entire Netty is asynchronous .
Book: The Definitive Guide to Netty
2.2, ZeroMQ
ZeroMQ is a multi-threaded network library based on message queues, which abstracts the underlying details of socket types, connection processing, frames, and even routing, and provides sockets across multiple transmission protocols. ZeroMQ is a new layer in network communication, between the application layer and the transport layer (divided according to TCP/IP), it is a scalable layer that can run in parallel and dispersed among distributed systems.
ZeroMQ is positioned as: a simple and easy-to-use transport layer, a socket library like a framework, which makes socket programming simpler, more concise and more performant. is a message processing queue library that scales elastically across multiple threads, cores and mainframes. The explicit goal of ZMQ is to "become part of the standard network protocol stack and then into the Linux kernel".
3. Worker internal communication technology (Disruptor)
3.1. The origin of Disruptor
- The relationship between a company's business and technology can generally be divided into three stages. The first stage is to follow the business. The second stage is the driving business stage that took several years to reach. In the third stage, technology leads the development of business and even the development of enterprises. So when we learn the technology of Disruptor, we have to mention the organization LMAX, because the technology of Disruptor is developed and open sourced by LMAX Company.
- LMAX is a foreign exchange gold exchange registered in the UK and regulated by the FSA (regulatory number 509778). LMAX is also the first and only top European financial company in Europe that adopts the Multilateral
Trading Facility (MTF) to have an exchange license and a broker license - LAMX has the fastest trading platform and top technical support. LMAX Exchange uses the "(MTF) Disruptor" technology, which can process orders in a very short time (generally within one 3 million seconds), and process 6 million orders per second in one thread . All orders are in the form of matching transactions, without exception. The Multilateral Trading Facility (MTF) was used to design the London Stock Exchange, Deutsche Borse and Euronext.
- In 2011, LMAX won the Best Trading System Award in the Financial Industry Technology Selection Competition and the Oracle "Duke Cup" Innovative Programming Framework Award for this technology.
3.2. What is Disruptor
1. Simple understanding: Disruptor is a Queue . Disruptor implements the function of "queue" and is a bounded queue (limited length). The application scenario of the queue is naturally the "producer-consumer" model.
2. There are many implementation classes of Queue in JDK, including but not limited to ArrayBlockingQueue and LinkBlockingQueue. The two underlying data structures are arrays and linked lists respectively. Array query is fast, linked list addition and deletion are fast, and it can adapt to most application scenarios.
3. But ArrayBlockingQueue and LinkBlockingQueue are all thread-safe. When it comes to thread safety, there will be keywords such as synchronized and lock, which means that the CPU will fight .
4. Disruptor is a lock-free exchange of information between threads (using CAS (Compare And Swap/Set) operations).
3.2. Main features of Disruptor
1. No competition = no lock = very fast.
2. The implementation of all visitors recording their own serial numbers, allowing multiple producers and multiple consumers to share the same data structure.
3. The serial number (ring buffer, claim strategy, producer and consumer) can be tracked in each object, plus the magic cache line padding, which means that there is no false sharing and unexpected competition.
3.3. Disruptor core technology point
Disruptor can be regarded as an event monitoring or message mechanism. In the queue, producers put messages in, and consumers take out and process them in parallel.
The bottom layer is a single data structure: a ring buffer.
Each producer and consumer has an order counter to show how the current buffering works.
Each producer-consumer can operate its own sequence counter and can read the other's counter, and the producer can read the consumer's counter to ensure that it is writable without locks.
The core component
Ring Buffer is a ring buffer, responsible for storing and updating the data (events) exchanged through the Disruptor.
Sequence numbers and manages the data (events) exchanged through them through sequentially increasing sequence numbers, and the processing of data (events) always increases one by one along the sequence numbers.
The bottom layer of RingBuffer is an array , and the order calculator is a 64bit long integer, which grows smoothly.
The running process of Ring Buffer:
1. Receive data and write it to the position of footer 31, and then write along the serial number, but will not bypass the footer where the consumer is located.
2. Joumaler and replicator read the position of 24 at the same time, they can read data in batches to 30
3. The consumption logic thread has read the position of 14, but cannot continue reading, because his sequence is paused at the position of 15, it needs to be Wait until his sequence gives him the serial number. If the sequence works properly, 30 data can be read.
Summarize:
Some of the above content only needs to be understood. The purpose is to better understand the principle of communication within Storm and lay a solid foundation for the future advanced road.