Reprinted from: http://skywalking.apache.org/zh/blog/2019-02-24-skywalking-pk-pinpoint.html
Author: Zhenfei, write to: 2019-02-24 Description : This article is written by individuals, version vested in the author, represent the personal views, for reference purposes only and do not represent the official views of skywalking. Note : This comparison is based on skywalking-6.0.0-GA and Pinpoint-1.8.2 (the latest version as of 2019-02-19). In addition, our technical selection directly denied Zipkin this time. The biggest reason is that it is intrusive to the code, and the same is true for CAT. This is completely unacceptable to us.
Skywalking uses Java dynamic bytecode technology
This should be the two best open source APM products at present, and both products have achieved no intrusion into the code through bytecode injection . Their comparison information is as follows:
OAP description : skywalking 6.x only has the concept of OAP, skywalking 5.x is called collector.
Next, conduct an in-depth analysis and comparison of each PK item. For more exciting and first-release content, please pay attention to the official account : [ Afei's Blog ].
Community comparison
At this point, skywalking will definitely win. On the one hand, skywalking has entered Apache incubation, and the community is quite active. And the project sponsor is Chinese, we can enter the official group (Apache SkyWalking exchange group:) to communicate 392443393with the project sponsor Wu Sheng, many questions can get everyone’s help as soon as possible (everyone who has played open source knows this value How big is it). And Pinpoint was developed by Koreans, and communication barriers are unavoidable. As for the commit frequency on github in the last year, skywalking and Pinpoint are on par, both close to 20:
Therefore, in the community, skywalking is even better.
Support language comparison
Pinpoint only supports Java and PHP, while skywalking supports 5 languages: Java, C#, PHP, Node.js, Go. If the company's services involve multiple development languages, then skywalking will be your better choice. And, if you want to implement your own probes (such as python language), the secondary development cost of skywalking is also lower than Pinpoint.
Note: There are developers on Github who have contributed support for Node.js to Pinpoint. Please click the link: https://github.com/peaksnail/pinpoint-node-agent. But it has stopped maintenance and hasn't been updated for several years!
Therefore, skywalking is even better in terms of language support .
Protocol comparison
SkyWalking supports gRPC and http, but it is recommended to use gRPC. Skywalking 6.x version no longer provides http mode (but it will still retain the 5.x data), and will consider deleting it in the future. And Pinpoint uses the thrift protocol. The agreement itself is no good or bad.
Storage comparison (important)
The author believes that storage is the biggest difference between skywalking and Pinpoint, because the underlying storage determines the upper layer functions.
Pinpoint only supports HBase, and the expansion cost is relatively high. This means that if you choose Pinpoint, you must have the ability to hold a set of HBase clusters (daocloud switched from Pinpoint to skywalking because the maintenance cost of HBase is a bit high). In this regard, skywalking supports a lot more storage. In this case, you can select the appropriate storage according to the technical characteristics of the team when selecting the technology, and you can also expand it yourself (but most of the production environment should be based on es storage).
Another defect that Pinpoint only supports HBase is that HBase itself has limited query capabilities (HBase can only support three query methods: RowKey exact search, SCAN range search, and full table scan), which limits Pinpoint's query capabilities, so the supported queries must be It is based on time (Pinpoint uses the mouse to delineate a time range to view the trace information within this range). Skywalking can be queried in any combination of multiple dimensions, such as time range, service name, Trace status, request path, TraceId, etc.
In addition, both Pinpoint and skywalking support TTL, which is a historical data retention strategy. Skywalking is configured in the application.yml of the OAP module to specify the retention time. And Pinpoint is realized through the ttl function of HBase. Through the hbase script provided by Pinpoint, https://github.com/naver/pinpoint/blob/master/hbase/scripts/hbase-create.hbaseyou can see: ApplicationTraceIndex is configured TTL => 5184000, SqlMetaData_Ver2 cooperates TTL => 15552000, and the unit is seconds.
Explanation : es does not completely crush HBase, es and HBase are not absolutely good or bad. es is strong in retrieval ability, but storage capacity is weak (below 100 billion, es is still fully capable of holding). HBase is strong in storage capacity but weak in retrieval. If the amount of logs collected is very large, es storage is more difficult. Of course, there is no crappy middleware, only crappy programmers, whether it is es or HBase, tuning is the most critical. Similarly, if you have certain requirements for retrieval capabilities, then HBase will definitely not be able to satisfy you. So, it's time to decide based on your business and needs again. Trade-off is really ubiquitous.
UI comparison
Pinpoint's UI is indeed slightly better than skywalking, especially the service topology display. But daocloud customized a UI for skywalking based on Pinpoint's style. Please poke link: https: //github.com/TinyAllen/rocketbot, project description is: rocketbot: A UI for Skywalking. The screenshot is shown below; therefore, Pinpoint is better if you compare only the native UI.
Scalability comparison
It seems that Pinpoint did not consider scalability at the beginning of the design, whether it is the underlying storage or the implementation of custom probes. One of the core design goals of skywalking is Pluggable , which is pluggable.
Take storage as an example. Pinpoint does not consider scalability at all. If skywalking wants to customize a set of storage, it only needs to define a class to implement an interface org.apache.skywalking.oap.server.library.module.ModuleProvider, and then implement some DAOs. As for Pinpoint, it has not considered expanding the underlying storage at all.
Take the implementation of a probe of its own as an example (for example, I want to implement a probe in the python language). Pinpoint chooses thrift as the data transmission protocol standard, and in order to save the data transmission size, try to use the data reference dictionary when transferring constants. Pass a number instead of passing a string directly, etc. These optimizations also increase the complexity of the system: including the difficulty of using the Thrift interface, the problem of UDP data transmission, and the registration of the data constant dictionary, etc. It is evident that Pinpoint has only supported Java and PHP since it has been in development for so many years. The data interface of skywalking has many standards and supports the OpenTracing protocol. In addition to the official support for Java, the support for C#, PHP and Node.js are all developed and maintained by the community.
There are also warnings mentioned later, the scalability of skywalking is much better than that of Pinpoint.
Finally, both Pinpoint and skywalking support plug-in development. Reference for Pinpoint plug-in development: http://naver.github.io/pinpoint/1.8.2/plugindevguide.html. Skywalking plugin development reference: https://github.com/apache/incubator-skywalking/blob/master/docs/en/guides/Java-Plugin-Development-Guide.md.
Therefore, skywalking is even better in terms of scalability .
Alarm comparison
Both Pinpoint and skywalking support custom alarm rules.
But the annoying thing is that if Pinpoint wants to configure alarm rules, it also needs to install MySQL (users, user group information and alarm rules when configuring alarms are persisted in MySQL), which leads to higher maintenance costs for Pinpoint. Both HBase and MySQL must be maintained.
Pinpoint支持的告警规则有：SLOW COUNT|RATE, ERROR COUNT|RATE, TOTAL COUNT, SLOW COUNT|RATE TO CALLEE, ERROR COUNT|RATE TO CALLEE, ERROR RATE TO CALLEE, HEAP USAGE RATE, JVM CPU USAGE RATE, DATASOURCE CONNECTION USAGE RATE。
Pinpoint periodically checks the data of the past 5 minutes every 3 minutes, and if there is an alarm that meets the rules, it will send sms/email to all users under the user group. It should be noted that the logic of sending sms/email needs to be implemented by itself, and Pinpoint only provides an interface com.navercorp.pinpoint.web.alarm.AlarmMessageSender. And when Pinpoint finds that the alarm persists, it will increment the time interval of sending sms/email by 3min -> 6min -> 12min -> 24min to prevent sms/email from frantically flushing.
Pinpoint alarm reference : http://naver.github.io/pinpoint/1.8.2/alarm.html
The skywalking configuration alarm does not need to introduce any other storage. Skywalking can configure alarm rules in config/alarm-settings.xml, and the alarm rules support customization.
The alarm rules supported by skywalking (the name in the configuration item is indicator-name) are: service_resp_time, service_sla, service_cpm, service_p99, service_p95, service_p90, service_p75, service_p50, service_instance_sla, service_instance_resp_time, service_instance_cpm, endpoint, endpoint_avg_p, endpoint_avg_p99, endpoint_la endpoint_p90, endpoint_p75, endpoint_p50.
Skywalking remotely calls the alarm notification service address defined in the configuration item webhooks through HttpClient. Skywalking also supports the silence-period configuration. Assuming that an alarm is triggered at TN, then the alarm will not be sent repeatedly during TN -> TN+period.
skywalking alarm reference : https://github.com/apache/incubator-skywalking/blob/master/docs/en/setup/backend/backend-alarm.md. Currently, only the metrics of Service, Service Instance, and Endpoint scope in the official_analysis.oal script are supported. The metrics of other scopes need to be expanded later.
Both Pinpoint and skywalking support common alarm rule configuration, but skywalking uses webhooks to be more flexible: SMS notifications, email notifications, and WeChat notifications are all supported. And Pinpoint can only send SMS/email notifications, and also needs to introduce MySQL storage, which increases the complexity of the entire system. Therefore, skywalking is even better in terms of warning .
JVM monitoring
skywalking supports monitoring: Heap, Non-Heap, GC (YGC and FGC). The main indicators that Pinpoint can monitor are: Heap, Non-Heap, FGC, DirectBufferMemory, MappedBufferMemory, but there is no YGC. In addition, Pinpoint also supports the function of viewing multiple indicators at the same time point. As shown below:
Therefore, Pinpoint is even better for JVM monitoring.
Service monitoring
Including the monitoring of operating systems and deployed service instances. The dimensions supported by Pinpoint are: CPU usage, Open File Descriptor, data source, number of active threads, RT, TPS. The dimensions supported by skywalking are: CPU usage, SLA, RT, CPM (Call Per Minutes). Therefore, the two are evenly matched in this regard, and there is no obvious gap.
Tracking granularity comparison
Pinpoint does a very good job in this regard, and the tracking granularity is very fine. As shown in the figure below, it is the trace information of Pinpoint on an interface:
The trace information of the same interface skywalking is shown in the following figure:
Remarks : This screenshot shows the plugin loaded by skywalking
apm-spring-annotation-plugin-6.0.0-GA.jar(this plugin allows tracking of bean methods in the spring context annotated with @Bean, @Service, @Component and @Repository).
Through comparison, it is found that Pinpoint is better in terms of tracking granularity .
Filter tracking
Both Pinpoint and skywalking can be implemented, and the configured expressions are based on the ant style. Pinpoint is configured on the Web UI filter wizard to customize the filtering and tracking. Skywalking can customize the filtering and tracking by loading the apm-trace-ignore-plugin plug-in. Skywalking is more flexible. For example, there are several services on a high-end server, in the common agent configuration file apm-trace-ignore-plugin. General filtering rules can be configured in config, and then personalized filtering can be configured for each service through -D.
Therefore, in terms of filtering and tracking, skywalking is even better .
Performance loss
Because Pinpoint collects too detailed information, it has the greatest loss in performance. The default strategy of skywalking is conservative and has little performance loss. Some netizens have done a stress test, the comparison is as follows:
Picture source: https://juejin.im/post/5a7a9e0af265da4e914b46f1
Therefore, in terms of performance loss, skywalking is even better .
Release package comparison
skywalking keeps pace with the times, all systems come standard with jar packages, and only need to execute the start.sh script for deployment. The collector and web of Pinpoint are still war packages, and they rely on web containers (such as Tomcat) for deployment. Please, it's 9012.
Therefore, skywalking is even better when it comes to releasing packages .
Support component comparison
Comparison of middleware supported by skywalking and Pinpoint:
- WEB container description : Pinpoint supports almost all WEB containers, including open source and commercial ones. Wkywalking only supports open source WEB containers, and does not support the two famous commercial WEB containers, Weblogic and Wevsphere.
- RPC framework description : For the support of RPC framework, skywalking kills Pinpoint in seconds. Even the niche motan and sofarpc are supported.
- MQ description : Skywalking supports a domestic MQ middleware RocketMQ more than Pinpoint. After all, RocketMQ is well-known in China, but it is common abroad. In addition, skywalking is also made in China.
- RDBMS/NoSQL description : Pinpoint supports both RDBMS and NoSQL slightly better than skywalking. For RDBMS, skywalking does not support MSSQL and MariaDB. For NoSQL, skywalking does not support Cassandra and HBase. As for H2, which is not supported by Pinpoint, it is not a problem at all. After all, the production environment will definitely not use H2 as the underlying storage.
- Redis client description : Although both skywalking and Pinpoint support Redis, skywalking supports three popular Redis clients: Jedis, Redisson, Lettuce. Pinpoint only supports Jedis and Lettuce. Once again, Pinpoint developed by Koreans ignores the Redis Client with the most stars on GitHub developed by Chinese at present.
- Log framework description : Pinpoint does not support log4j2? But someone has already developed related functions, please click the link for details: log4j plugin support log4j2 or not? https://github.com/naver/pinpoint/issues/3055
Through the comparison of skywalking and Pinpoint support middleware, we found that skywalking’s support for domestic software really kills Pinpoint in all directions, such as niche RPC frameworks: motan (produced by Weibo), sofapc, Ali’s RocketMQ, Redis customers Redisson, and elastic-job, a distributed task scheduling framework. Of course, it also reflects that the influence of domestic open source software in the world is still very small.
No one is good or bad in this regard, after all, each company uses a different technology stack. If you have a strong demand for RocketMQ, then skywalking is your best choice. If you have a strong demand for es, then skywalking is also your best choice. If HBase is your strong demand, then Pinpoint is your best choice. If MSSQL is your strong demand, then Pinpoint is also your best choice. In short, it all depends on your project.
to sum up
After comparing skywalking and Pinpoint in all directions, we found that for the two excellent APM softwares, there is a feeling of being good and bright. The advantages of Pinpoint are: a very fine-grained tracking data, a powerful user interface, and the massive storage capabilities brought by using HBase as storage. The advantages of skywalking are: a very active Chinese community, support for probes in multiple languages, very comprehensive support for domestic open source software, and powerful retrieval capabilities brought by using es as the underlying storage, and the scalability and customization of skywalking Transformation is better than Pinpoint:
- If you have massive log storage needs, Pinpoint is recommended.
- If you value the convenience of secondary development more, skywalking is recommended.
Finally, refer to the above comparison, combine your needs, which can not be compromised, which can be discarded, so as to better choose the most suitable APM software for you.
continue reading:
What is the end of the bytecode injection of Skywalking?