Tencent open source project once again usher in heavy, 14, Tencent announced an open source high performance computing framework Figure Plato, which is in just one week, the fifth major open-source projects.
Relative to other large scale map of the current view of computing framework worldwide, Plato can meet the computing needs of one billion nodes, the algorithm calculates the time from days to minutes, the performance overall lead ahead of other major distributed computing framework map and breaking the original often take hundreds of server resource bottlenecks, now only requires a minimum of ten servers to complete the calculation.
Tencent official said Plato team in the East China Sea: "Plato has supported include internal Tencent micro-letters, including many of our core business, in particular to provide support for all types of computing Tencent large scale social graph data to solve the existing framework can not be calculated in other limited resources and limited time to complete the calculation of difficulty .Plato not only created for Tencent tremendous business value of open source will continue to promote the coordinated development of Figure computing technology and industry, and accelerate innovation. "
In fact, a calculation of "map" does not refer to ordinary images and photos, but is used to represent an association relationship between objects abstract data structures, a calculation is expressed as a problem to map the data model and to be addressed process. Figure computing may be the integration of different sources, different types of data to be analyzed in the same graph, the results had been difficult to find independent analysis, thus becoming a social network, it is essential recommendation system, network security, text retrieval and bio-medical fields data analysis and mining tools.
Plato is a high performance computing framework diagram TGraph independent research and development team to integrate internal resources of Tencent internal map calculation, named Plato is a tribute to the great mathematician Plato, the current team is Tencent cloud big data package Plato, is about to open to all developers use.
It is understood that computing performance Plato aspect is extremely powerful, than most on the market-leading computing framework Spark GraphX also Figure 1-2 orders of magnitude higher, it will algorithm time from days to minutes, lifting the number of times the performance also marks a calculation entered the era of minutes; Another big advantage is, Plato is much smaller than the mainstream view of computing framework in terms of memory consumption, reduced 1-2 orders of magnitude than the Spark GraphX, only about 10 small and medium-scale servers cluster, to complete large scale map calculation, after hundreds of servers frequently need to limit the comparison, pressure on resources and computing costs have been greatly reduced.
Currently, Plato mainly provides two core capabilities: Figure in the order of calculation and offline map data in Tencent Tencent data indicate the order of learning. Meanwhile Plato natural fit Kubernetes, YARN and other resource scheduling platform and provide support for major file systems of a variety of interfaces, can provide a more friendly operating environment for developers.
The architecture, the core framework is Plato FIG adaptive calculation engine, it is possible depending on the type of FIG algorithm to provide more flexibility to the calculation mode selected for developers, including adaptive calculation model, and pipelining of shared memory computing model calculation mode . In addition, a good interface designed to support new computing access communication mode.
Plato overall architecture diagram
In the above calculation engine, Plato provides multi-level interfaces for algorithm designers or specific business: from the underlying API, to graph algorithms library, and then tailored "solutions" for the specific business - drawing toolset. By means of these interfaces and the application layer, can also be calculated offline Plato results with other machine learning algorithm are combined together to support different business top.
It is worth mentioning that the current map feature in Plato algorithm library, central node index, connected graph and community recognition and other algorithms have been open, the future will further open more algorithms.
Plato high-performance, scalable and simple-swap features broad application prospects in social networking, recommendation systems, biomedical and other fields, such as regular web pages to enhance the influence of ordering user's search experience, analysis of a large social network structure to accurate recommended to the user services to the development of more effective clinical medicine to understand protein interactions through the sub-matching, etc. FIG.
Since last year, 930 structure adjustment, the open source collaborative strategy to become one of the important technological development of Tencent, and led to heavy intensive projects of foreign revenue. Just Techo Developers Conference last week, Tencent announced TubeMQ, Tencent Kona JDK, TBase, TKEStack four key open source projects. With the open source Plato, Tencent added a big move in the open field. It is reported that Tencent has been open sourced on Github 89 projects, more than 1000 contributors involved in the open source contributions, with more than 260,000 Star number, contribute standings among the highest in the world Github company.
Plato Open Source Address: https://github.com/tencent/plato
