Real-time computing system---Storm

Reprinted from http://blog.csdn.net/hljlzc2007/article/details/12976211

Scenario
With the rapid development of information technology, information has shown explosive expansion, and people have more diverse and convenient ways to obtain information. The timeliness requirements are getting higher and higher. As an example in the search scenario, when a seller posts a baby information, of course, what he hopes is that the baby can be searched, clicked, and purchased by the seller immediately. On the contrary, if the baby has to wait until the next day or more. It took a long time to be found, and it is estimated that this eldest brother will be cursed. To give another recommended example, if a user bought a pair of socks on Taobao yesterday, and wants to buy a pair of swimming goggles to go swimming today, but finds that the system spares no effort in recommending socks and shoes to him, it is totally unacceptable for him who is looking for swimming goggles today. Turn a blind eye to the behavior, it is estimated that this buddy will want to recommend your sister. In fact, code farmers who know a little bit of background knowledge know that this is because the background system does full processing once a day, and most of it is done in the dead of night, so of course what you do during the day will be reflected tomorrow. La.

To implement a real-time computing
system , most of the data processing uses the well-known hadoop or hive. As a batch system, hadoop has been widely used in massive data processing due to its advantages of high throughput and automatic fault tolerance. However, Hadoop is not good at real-time computing, because it is naturally born for batch processing, which is also the consensus of the industry. Otherwise, real-time computing systems such as s4, storm, and puma would not have sprung up in the last two years. Leaving aside the systems of s4, storm, and puma, let's first take a look at what problems we need to solve if we design a real-time computing system ourselves.

Low latency. It is said that it is a real-time computing system, and the delay must be low.
high performance. Low performance is a waste of machines, and waste of machines is criticized.
distributed. Systems are all created for application scenarios. If your application scenarios, your data, and a single computer can handle it, then you don’t need to consider these complex issues. We're talking about situations where a single machine can't handle it.
Scalable. With the development of the business, our data volume and calculation volume may become larger and larger, so we hope this system is scalable.
fault tolerance. This is a common problem in distributed systems. A node down cannot affect my application.
Well, if you only need to solve these five problems, there may be countless solutions, and each has its own merits. Just pick one solution and use the message queue + worker processes distributed on each machine. Let's continue to look down.

Easy to develop applications on it. Dear, does the system you design require application developers to consider the distribution of various processing components and the transmission of messages? If it is, it's a bit of a hassle, and developers may not use it well and won't want to use it.
Messages are not lost. A baby message posted by a user can't be lost while it's being processed in real-time, right? To be more strict, if it is an application for accurate data statistics, then it should process more or less messages. This requirement is a bit high.
Messages are strictly ordered. There is a strong correlation between some messages, such as the update and delete operation messages of the same baby, if the order is messed up during processing, the effect will be completely different.
I don't know if everyone has their own answers to these questions. Let's take a look at storm's solutions with these questions.

What is Storm
If there is only one sentence to describe Storm, it might be this: a distributed real-time computing system. According to the author of Storm, the significance of Storm for real-time computing is similar to that of hadoop for batch processing. We all know that Hadoop based on google mapreduce provides us with map and reduce primitives, making our batch programs very simple and elegant. Likewise, Storm provides some simple and elegant primitives for real-time computing. We will introduce it in detail in Section III.

Let's take a look at the applicable scenarios of storm.

Stream data processing. Storm can be used to process a steady stream of messages, and then write the results to a storage after processing.
Distributed RPCs. Since the processing components of Storm are distributed and the processing latency is extremely low, it can be used as a general distributed RPC framework. Of course, our search engine itself is also a distributed RPC system.
After talking for a long time, it seems to be a very mysterious thing. Let's start to explain the basic concepts of Storm and some of its internal implementation principles.

The basic concepts of Storm
First, we understand the basic concepts of Storm through a comparison between Storm and hadoop.

Hadoop Storm
System Role JobTracker Nimbus
TaskTracker Supervisor
Child Worker
Application Name Job Topology
Component Interface Mapper/Reducer Spout/Bolt
Table 3-1

Next, let's look at these concepts in detail.

Nimbus: Responsible for resource allocation and task scheduling.
Supervisor: Responsible for accepting tasks assigned by nimbus, starting and stopping worker processes that are managed by themselves.
Worker: A process that runs specific processing component logic.
Task: Each spout/bolt thread in a worker is called a task. After Storm 0.8, tasks no longer correspond to physical threads. Tasks of the same spout/bolt may share a physical thread, which is called an executor.
The following figure describes the relationship between the above roles.
System structure
Figure 3-1

Topology: A real-time application running in storm, because the flow of messages between components forms a logical topology.
Spout: A component that generates source data streams in a topology. Typically spouts will read data from external data sources and convert them to source data inside the topology. Spout is an active role. There is a nextTuple() function in its interface. The storm framework will call this function continuously, and users only need to generate source data in it.
Bolt: A component in a topology that accepts data and then performs processing. Bolt can perform filtering, function manipulation, merging, writing to the database, and anything else. Bolt is a passive role, and there is an execute(Tuple input) function in its interface. After receiving a message, this function will be called, and the user can perform the operation they want in it.
Tuple: The basic unit of a message delivery. It should have been a key-value map, but since the field names of the tuple passed between the components have been defined in advance, you only need to fill in the values ​​in order in the tuple, so it is a value list.
Stream: Continuous transmission The tuple constitutes a stream.
10. Stream grouping: The partition method of the message. Storm provides several practical grouping methods, including shuffle, fields hash, all, global, none, direct and localOrShuffle, etc.

Compared with other real-time computing systems such as s4 and puma, the biggest highlight of Storm is its record-level fault tolerance and guaranteed Transactional capabilities for precise processing of messages. Let's focus on the realization principle of these two highlights.

The basic principle of Storm record-level fault tolerance
First , let's take a look at what is record-level fault tolerance? Storm allows users to specify a message id when emitting a new source tuple in a spout. The message id can be any object. Multiple source tuples can share a message id, indicating that these multiple source tuples are the same message unit to the user. Record-level fault tolerance in Storm means that Storm will inform the user whether each message unit has been fully processed within the specified time. Then what is called complete processing, that is, the source tuple bound by the message id and the tuple subsequently generated by the source tuple have been processed by each bolt that should be reached in the topology. for example. In Figure 4-1, tuple1 and tuple2 bound by message 1 in spout are processed by bolt1 and bolt2 to generate two new tuples, which eventually flow to bolt3. When the process has finished processing, message 1 is said to be completely processed.
Message Passing
Figure 4-1

has a system-level component in Storm's topology called acker. The task of this acker is to track the processing paths of several tuples bound to each message id flowing from the spout. If these tuples are not fully processed within the maximum timeout time set by the user, the acker will inform the spout that the message processing failed On the contrary, it will inform spout that the message processing was successful. In the description just now, we mentioned "recording the processing path of tuple". If you have tried this, you can think carefully about the complexity of this matter. But Storm does it in a very clever way. Before explaining this method, let's review a mathematical theorem.

A xor A = 0.

A xor B…xor B xor A = 0, where each operand occurs and only twice.

The clever method used in storm is based on this theorem. The specific process is as follows: in the spout, the system will generate a corresponding 64-bit integer for the message id specified by the user as a root id. The root id will be passed to the acker and subsequent bolts as the unique identifier of the message unit. At the same time, each time a new tuple is generated by spout or bolt, it will give the tuple a 64-bit integer id. After the spout emits the source tuple corresponding to a message id, it will inform the acker of the root id it emits and the ids of the generated source tuple. As for the bolt, each time it receives an input tuple and processes it, it will also inform the acker of the id of the input tuple it has processed and the id of the newly generated tuple. Acker only needs to perform a simple XOR operation on these ids to determine whether the message unit corresponding to the root id has been processed. The following is an illustration to illustrate this process.

Figure 4-1 Binding message 1 in spout generates two source tuples with ids 0010 and 1011 respectively

. Figure 4-2 When bolt1 processes tuple 0010, a new tuple is generated with id 0110.

Figure 4-3 Bolt2 processes When tuple 1011, a new tuple is generated with an id of 0111.

Figure 4-4 Bolt3 receives tuple 0110 and tuple 0111, and no new tuple is generated.

Some careful students may find that there is a possible error in the fault-tolerant process , that is, if the generated tuple ids are not completely distinct, the acker may incorrectly evaluate to 0 before the message unit is fully processed. This error does exist in theory, but in practice its probability is extremely low and can be completely ignored.

Storm's transaction topology
Transactional topology is a feature introduced in storm 0.7 and has been encapsulated as Trident in the recently released 0.8 version, providing a more convenient and intuitive interface. Due to space limitations, a brief introduction to the transaction topology is given here.

The purpose of transaction topology is to meet the scenarios with extremely strict requirements for message processing, such as calculating the number of transactions of a user in real time, and the results are required to be completely accurate, neither more nor less. Storm's transaction topology is completely implemented based on its underlying spout/bolt/acker primitives, and an elegant implementation is obtained through a layer of clever encapsulation. Personally, I think this is one of the biggest charms of Storm.

The transaction topology is simply to divide messages into batches. Messages in the same batch and messages between batches can be processed in parallel. On the other hand, users can set certain bolts as committers, and Storm can Ensure that the committer's finishBatch() operations are executed in strictly non-descending order. Users can take advantage of this feature to achieve precise message processing through simple programming skills.

Storm is on Taobao.
Because the kernel of Storm is written in clojure (but most of the expansion work is written in java), it brings some difficulties for us to understand its implementation. Fortunately, storm is relatively stable in most cases, of course We are also trying our best to familiarize ourselves with the world of clojure. When we use storm, we usually choose java language to develop applications.

In Taobao, storm is widely used for real-time log processing, which appears in real-time statistics, real-time risk control, real-time recommendation and other scenarios. Generally speaking, we read real-time log messages from kafka-like metaQ or hbase-based timetunnel, and after a series of processing, finally write the processing results to a distributed storage for application access. Our daily real-time message volume ranges from millions to billions, and the total amount of data reaches terabytes. For us, Storm is often used with distributed storage services. In our ongoing real-time analysis project of personalized search, we use the architecture of timetunnel + hbase + storm + ups to process billions of user log information every day, and the delay from the occurrence of user behavior to the completion of analysis is in seconds.

Storm's future
version of Storm 0.7 series has been widely used by major companies. The recently released version 0.8 introduced State, which has evolved from a pure computing framework to a new real-time computing tool that includes storage and computing. There is also the Trident mentioned just now, which provides a more friendly interface. At the same time, the characteristics of the customizable scheduler also provide a more convenient means for optimization for different application scenarios. Some people are already using Storm-based real-time ql (query language) Step on the script. In terms of service, Storm has been working towards integrating into the mesos framework. At the same time, Storm is also constantly optimizing the implementation details, using many excellent open source products, including kryo, Disruptor, curator and so on. It is conceivable that when storm develops to version 1.0, it must be a very outstanding product, let us wait and see, of course, it is better to participate in it, students.