For distributed online services, a request needs to go through multiple modules in the system, and hundreds of machines cooperate to complete a single request. A typical scenario is a search engine user retrieval, and manpower alone cannot grasp the performance overhead of each stage of the entire request. , and cannot quickly locate the performance bottleneck in the system. The Google Dapper article describes Trace Infrastruce—Dapper, which is widely used in Google's internal services (see the original address here , and the translation address here ). The article itself is very easy to understand, and there is no complicated and sophisticated implementation mechanism (it seems to be published by g company. The characteristics of the article), programmers with some experience in distributed online services can understand it well (English version), here are only some points to record. Zipkin is a Trace system component open sourced by Twitter. Google Dapper is referenced in its implementation. See http://twitter.github.io/zipkin/ for the project homepage .
Google Brave
Goal: ubiquitous deployment, and continuous monitoring, only ubiquitous and continuous monitoring can ensure that all problems can be tracked, because the service is also 7*24. In order to achieve these two points, Dapper has split the following goals for this Tracing component.
- Low overhead: Distributed online systems have strict performance and resource requirements. The Trace component must have a small impact on the service, otherwise it will not be enabled.
- Application-Level Transparency: Application-level developers do not need to care about the Trace component. Trace is embedded in the basic general library to provide high stability. If Trace needs the cooperation of application developers, it may introduce additional bugs Makes the Trace system more vulnerable.
- Scalability: The number of online clusters of Google's services can be imagined, and a single retrieval may span hundreds or even thousands of machines, so the scalability of this Trace Infrastructure is also very important.
- Analysis Quickly: Quickly export and analyze data, which can quickly locate system abnormalities, respond in time, and improve system stability.
Typical retrieval scenario: A single user retrieval runs through multiple modules at the front and back ends, and rpc is used to communicate between the modules, as shown in the figure below. The request requires the communication and cooperation of multiple modules composed of different teams and different languages in the system. A typical one is from the top And the following call sequence. Dapper's collection method is based on the tree formed by this call sequence, which is split into different spans (one rpc remote call), which are strung together through the globally unique Trace Id, and the span points to its own parent span.
Practical experience of Trace components:
- Low sampling rate: In the case of ensuring data error, Dapper uses a low sampling rate as much as possible to ensure the low overhead of Trace. On the one hand, it will leave more performance space for the application layer's own Annotation tag, and it can also be stored on the hard disk. Save data for longer periods of time.
- Out-of-band data collection (out-of-band): The Trace component implements the span information to the local machine, and uses the daemon process for out-of-band collection, which can prevent the trace data from occupying the retrieval bandwidth. At the same time, not all calling processes are perfectly nested, and some components will report some results to the upstream before returning the results downstream, which is not suitable for binding Trace results with retrieval data.
- Data privacy: The Trace system is only for system performance overhead diagnosis. Considering privacy, Dapper only records the RPC name and performance overhead. For some policy and business data, you can consider using a system like Online Debug to implement permission control.
- Simplified basic lib: The basic code base of Dapper is very simplified, which is easy to implant in various general libraries to ensure stability and performance overhead. It is implanted by default on most of the basic libraries of control flow to record the overhead of each span. Among them, C++ The code implementation is less than 1000 lines.
- Controlling the resource consumption of the Dapper data collection daemon: Some routine tasks on the machine, such as data statistics script running, automatic update of external data, etc., will also have a performance impact on the processes deployed on the machine. Therefore, the performance data collection of the Dapper daemon process is very cautious about the network and CPU used, and the Dapper daemon process is the lowest priority of the kernel scheduler.
- Open API and easy-to-use UI: convenient DAPI to access the collected data imported into Bigtable, other users can use the API to build web applications, executable commands, etc. (from experience, RESTFul is a good choice), and Provides a nice UI by default for easy querying.
Twitter Zipkin
Zipkin's project homepage says - Zipkin is a distributed tracing system that helps us gather timing data for all the disparate services at Twitter. It manages both the collection and lookup of this data through a Collector and a Query service. We closely modelled Zipkin after the Google Dapper paper . Looking at Zipkin's Architecture document, the description also implements some of the core descriptions in the Dapper article.
Zipkin Infrastructure
- The collection system of Trace data uses Facebook's open source Scribe as the log transmission path to transmit Trace data on different services to Zipkin/Hadoop.
- Zipkin's Storage system first used Cassandra, and later added support for Redis, HBase, MySQL, PostgreSQL, SQLite, and H2 storage systems. It is speculated that this is also related to Twitter's abandonment of Cassandra and returning to MySQL.
- A Java Async RPC service called Finagle is widely used in Twitter, and embedding the trace library into this Finagle became a natural option to support system tracing. This Library is also based on several concepts in Dapper's article (Annotation - key/value including host and timestamp; Span - a specific rpc call, including multiple Annotations; Trace - composed of a set of spans, sharing a root span) .
- When the Client and Server communicate, TraceId (identifying the entire trace), SpanId (independent RPC request), optional Parent SpanId (a service will call multiple downstream RPC services) and Sample boolean variables will be added to the tracing header information. (Identifies whether to collect this time).
- Provide data access and query functions through Thrift Api, and build your own UI (based on Rails, using d3js).
According to the description on the homepage, after the Server receives the Client request, it parses the processing logic flow chart of the Trace header, and draws a simple flow chart by itself (the words are ugly--).
Whether it is Dapper's article or Zipkin's homepage, the description of Trace Infrastructure in distributed systems is relatively simple, only mentioning some necessary basic components and the issues that each component needs to pay attention to. However, combined with practical work experience, such a complete Trace system is very helpful for abnormal diagnosis and performance optimization of distributed online services, and such a Trace component has very high requirements on each basic component. Robust and complete basic components are very beneficial for the construction of such a system.
http://www.tuicool.com/articles/F3YNrm