I do not know how to deal with the kind of problems mentioned at the beginning of it? The most simple and crude way is to focus on personnel data into a conference room to face, how to do?
Client Developer: I checked the logs, the client process requests a share 5s, the request is initiated from a few minutes and seconds, you check under the server's log;
Trading Systems Developer: I was here a few seconds some sort of request received, the trading system had spent more than 4s, some of which call payment gateway spent nearly 4s, look over there gateway log it;
Gateway developers: here is some sort of request I received a few seconds, the gateway spent a total of a little more than 3s, most of the time spent on call to a third party;
Estimated that most people are beginning to deal with such problems so simple and crude. But if you come every few days so all of a sudden you stand it? You every day hundreds of thousands of data allows you to order number, you can take the time to write code? Estimate toilet even paid on time is gone now. This last problem could spread to the leadership there, leading to a global report data generally like, how you give him out of this report? Is not helpless, suddenly I want to change a little work, ha ha. We really received such a demand, a bunch of people in there then there is no then the awk.
"When the thing become a norm, that means we might need a tool to liberate himself, by the people, after all, it is unreliable", in this context we decided to introduce a call chain tracking tool to liberate we, that is today's hero jaeger. Jaeger instructions on the Internet a lot to learn about recommended Quguan network system https://www.jaegertracing.io , I came out here just to share some ideas on the building process and to use and share with everyone.
jaeger architecture
The introduction of a direct view of the official website
jaeger Introducing:
jaeger-client: jaeger clients to achieve opentracing agreement;
jaeger-agent: jaeger client to an agent, client call chain to the collected data to the agent, then the agent sent by the Collector;
jaeger-collector: call chain is responsible for receiving data jaeger client reports jaeger agent or up, then do some checking, such as the time range of the legality, and ultimately will be processed stored in the internal memory to the rear end;
jaeger-query: call a dedicated query service chain, it has its own independent UI;
jaeger-ingester: Chinese name "person feeding", and can be used to read data written from the back-end storage kafka jaeger, such as Cassandra and elasticsearch;
spark-job: spark-based computing tasks, dependencies and services can be calculated, the number of calls and so on;
Which jaeger-collector and jaeger-query is a must, the rest are optional, we did not use the agent to report the way, but let the client endpoint to report directly to the collector.
Build jaeger
Because our application services are based on the deployment of the container, so our jaeger service also follows the style of the past.
docker start jaeger-collector
docker run -d --rm -p 14268:14268 -p 14269:14269 -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://10.200.46.229:9200 jaegertracing/jaeger-collector:1.11
docker start jaeger-query
docker run -d --rm -p 16686:16686 -p 16687:16687 -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://10.200.46.229:9200 jaegertracing/jaeger-query:1.11
Application Access
The next step is how to get access on each end of the call chain, and here only need to grasp a principle like, "try to make the access side without perception, not invasive," where simply our way of access:
- Client access: The client uses an access way okhttp interceptor, the use of trace context transfer request header, there may also be fit together and EventListener okhttp get some network level indicators, such as dns resolution time, connection initiation time and the like;
- access to web application: web end use springmvc access interceptor embodiment, the first http request from the inside to extract trace context based on the context and construct a span of springmvc, remember the end of the finish Austrian request or the call data may be longer chain such:
- How to integrate the frame RPC: RPC framework will generally provide extension points allow users to do some things integrated frame, it can be used to take dubbo Filter parameter passing and implicit way to achieve pass the request context;
- How to integrate external calls: Some calls are based sdk or httpclient call, how do we call this type of implant logical chain call it? Here AspectJ have to admire the powerful, in order to avoid detours you I will recommend you to look at the "spectj-maven-plugin" the maven plugin, what? Spring is not based on that pile of notes on it yet, why the introduction of maven to do the job. You also need to look at the estimated runtime and compiler implants implanted concepts and application scenarios.
In particular you should Span packaged into what kind of depend on you free to play, but not too far off the mark, recommended that reference this https://opentracing.io/docs/overview/spans/ .
online
Ask yourself a few questions before the line, I wrote interceptor whether robust, Throws the call will not affect the normal right? The need to assess the amount of data? Do not put a line on the back-end storage and killed.
Use jaeger-quey to retrieve the call chain
- First select a service and then do some complex searches for this service, such as for a label, and other time-consuming operations;
2. If you have the right data to meet the conditions will demonstrate results
The above figure shows two calls were paid links, a success, a failure, you might ask: jaeger is how to judge the success of failure? It is simply a special label, direct training and preparation of the document read to you a opentracing understand https://github.com/opentracing/specification/blob/master/semantic_conventions.md .
3. Review the call chain details
4. Check dependencies, and the number of calls
Maybe you also setting up the service, call chain data can see, but they do not see the call graph, do not worry you to walk around it to know https://www.jaegertracing.io/docs/1.11/ faq / .
Well, today went to this big Saturday night set aside some time to sort out what recent work, you also want to have a little bit of help.