Bit real-time calculation road of development and platform architecture practice

--- toc: truetitle: pieces of real-time computing platform for the development of road infrastructure and practice date: 2019-08-25tags:

Flink
Big Data
---

Didi's core business is a real-time online service, and therefore has a wealth of real-time data and real-time computing scene. This article describes the development of the road and drops in real-time platform architecture of computation.

Real-time computing evolution

With the development of business and pieces, and pieces of real-time computing architectures are rapidly evolving. To date probably gone through three stages:

Self-built business side small cluster stage;
Centralized large clusters, platform stage;
SQL stage.

The following illustration identifies an important milestone which will be given later elaborated:

In 2017, it drops and there is no unified real-time computing platform, but each business side of self-built small clusters. Which used engines Storm, JStorm, Spark Streaming, Samza and so on. Self-built business side there is a small cluster mode drawbacks:

We need to pre-purchase a large machine, since a single business monopoly, resource utilization is usually relatively low;
The lack of effective monitoring alarm system;
Maintenance is difficult, requires a lot of energy involved in the business side to guarantee the stability of the cluster;
The lack of effective technical support, and each precipitate something difficult to share.

In order to effectively solve the above problem, and pieces from the beginning of 2017 began to build a unified platform and real-time computing clusters.

Technology selection, we chose the pieces based on the current situation inside with large-scale data cleaning Spark Streaming engine, while the introduction of On-YARN mode, and build the authentication, authorization, resource isolation, billing YARN use of multi-tenant system, etc. mechanism.

With respect to off-line computing, real-time computing tasks have higher requirements for stability, for which we construct two resource isolation system:

The first layer is based on CGroup do process (Container) level of CPU and memory isolation;
The second layer is a physical machine level of isolation.

We have to support Node Label YARN through the transformation of FairScheduler. Effect achieved as shown below:

Task ordinary business of a mixed run on the same machine Label, and special mission operations run on a dedicated machine Label.

Through a centralized platform for the construction of large clusters and virtually eliminate the evils of self-built business side of small clusters, real-time computing has entered the second stage.

With the development of business, we have found Spark Streaming of Micro Batch mode on some low latency alarm business and online business become stretched. So we introduced based Flink Native Streaming mode as a new generation of real-time calculation engine.

Flink can be done not only millisecond delay, but also provides a wealth of window function based on Process Time / Event Time. Flink based on our joint business framework of the party's largest business gateway drops flow monitoring system, and support rapid change notification such as passenger position, trajectory anomaly detection and other online services.

Real-time computing platform architecture

For maximum convenience of business development and management side stream computing tasks, we constructed a real-time computing platform as shown:

Provided in the flow calculation based on the engine StreamSQL IDE, alarm monitoring, diagnostic systems, kinship, task management and control capabilities. Their role is as follows:

StreamSQL IDE. Hereinafter will be described, is a Web of SQL IDE;
Monitoring alarm. Providing task level of survival, delay and flow monitoring and alarm capabilities based monitoring;
Diagnostic system. Including flow curve graph view, Checkpoint, GC, and other resources, as well as real-time log retrieval capabilities.
blood relation. We built reporting capability in blood flow calculation engine, thus showing kinship with the task flow upstream and downstream on the platform;
Mission control. To achieve the task to submit multi-tenant system, start and stop, asset management and other capabilities. Web tasks submitted by eliminating the traditional client mode, making the platform fully controllable entrance, built in the following versions optimized to quickly on-line.

Real-time construction rule matching service

Inside drops a large number of real-time operational scenarios, such as "city after a passenger bubble 10 seconds no orders." For dependencies between detection of such an event scene, with CEP Fink it is very appropriate.

But the community does not support the version of the CEP description languages, each rule need to develop an application, but does not support dynamic update rules. To solve these problems, and pieces done a lot of extensions and optimization. The main function extensions changes are:

Support wait operator. For example, just in the operating rules, the community version is not expressed. And pieces by increasing the wait operator to achieve this kind of demand;
DSL language support. Groovy-based analytical engine and Aviator, we achieved DSL capability described rules shown below:

Single-tasking multi Rules of dynamically updated. Since the operator to configure real-time operating rules by the students line, so the number of rules, regulations and rules of the content life cycle will change frequently. Each rule this case an application is not very realistic. To this end we have developed a multi-mode and regular support dynamic updates.

In addition to the functions of development, large-scale response to the challenges of operating rules, and pieces also done a lot of optimization on the CEP performance, are:

SharedBuffer reconstruction. Flink MapState reconstruction SharedBuffer, each reduced-state interaction based data processing. While peeling rules and user data greatly reduces the amount of data each time from the matching state deserialized;
Increase access cache (Community contribution already). SharedBuffer cache each time data processing needs to be updated reference count, delay the update;
A simplified event time semantic processing. In the key is to avoid the dispersion of all the key data to traverse each time updating watermark;
Multiplexing conditionContext (Community contribution already). PartialMatch repeated inquiries to the elements while reducing the query conditions.

CEP will optimize performance over several orders of magnitude. With extensions, we offer a service model as shown in the interior pieces:

Only needs cleaning business data and provide real-time list of rules can have the ability to match the API is responsible for the rules.

Currently on the floor drops CEP has been operating in the express personalized, real-time detection of abnormal work orders and other services, and achieved good results.

StreamSQL construction

Just as in the offline calculation of the Hive, like MapReduce, Streaming SQL is the inevitable trend of development. Can greatly reduce the difficulty of developing the business side of the stream computing by SQL, business parties no longer need to learn Java / Scala, you do not need to understand the details of execution and various engine parameters tuning.

To this end we launched StreamSQL construction project in 2018, expanding the capacity of community Flink SQL basis:

Extended DDL syntax. As shown below, the open interior of the main bit message queue and real-time storage systems (built StreamSQL open message queue and store Embodiment):

By parsing capability built common message formats (such as json, binlog, standard log), so that users can easily write DDL syntax, and to avoid duplication write format parsing statement.
Expand the UDF. Bit internal processing logic for common, many built UDF, including string manipulation, date handling, Map object processing, spatial location of treatment.
Support split syntax. A single input multiple output streams at source inside the drops are very common, and we transformed the Calcite to support diversion semantics.
It supports TTL-join semantics. Window Join conventional window boundary data because of the mutation, can not meet the needs of the interior of the drops. To this end we have introduced a TTL State, and based on this developed based on TTL Join Shuangliu join and dimension tables join.
StreamSQL IDE. After the previously mentioned platform we do not provide the client, but submission and control tasks via the Web. Therefore, we have developed a corresponding StreamSQL IDE, enables the development StreamSQL on the Web, while providing a syntax check, DEBUG, diagnostic capabilities.

Currently StreamSQL has successfully landed in pieces, stream computing development costs are significantly reduced. Expected future will bear 80% of the traffic flow calculation.

to sum up

As a travel in the field of Internet companies, and pieces have a natural demand for real-time computing.

Over the past year or so, we have constructed a centralized real-time computing platform from zero, changing the business side of a small cluster of self-built situation. To meet the demand for low latency business, successfully landed Flink Streaming, and built in real time matching rule (CEP) services and StreamSQL based Flink, so that stream computing development capabilities significantly reduced. The future will further expand StreamSQL, batch and flow field in uniform, IoT, real-time machine learning and other exploration and construction.

follow me

Micro-channel public number: Zhisheng

In addition, I myself put together some Flink learning materials, has put all public micro-channel number (zhisheng), you can reply Keywords: Flink can unconditionally get to. Alternatively, you can add me to believe you can add my micro-micro letter: yuanblog_tzs , explore the technology!