Secret Tencent worker bees: enterprise code management Collaboration Solutions
Internet agile development is inseparable from the efficient code management systems. As part of the foundation of the development process, code management with upstream and downstream development of the role of series link demand management, continuous integration, continuous delivery, but also carries the corporate pursuit of code quality, encourage the construction of cultural code reuse and other engineers. Tencent has nearly 30,000 R & D personnel, product lines long, wide range of business, different team size, technology stack and R & D R & D collaboration models are put forward different demands, but also led to the code base size and uneven development process. At the same time build system, distribution system and so need to check out all the code, the higher the degree of automation, pressure to access the code base will be. The code provides a safe and stable service, management of different sizes code repository, support various types of R & D processes, are the three major challenges facing the management of code. Based on industry conditions and their own development needs, Tencent chose Git Git-based system, within the hatched since the inquiry - the worker bees.
We must first solve the server-side code libraries store expansion issue, because the single storage node TB storage capacity can not meet the level of growth, there are custom data fragmentation and common distributed file storage are two scenarios to consider. The advantages of distributed storage is the application layer shields the underlying storage structures, architecture is relatively simple, but the code management IO intensive applications, reliance on the performance of the distributed file system IO, portability is not strong. Instead custom data slice partitioning strategy can be freely controlled, flexible resource load balancing, on the underlayer additionally stores each slice, distributed storage can also be combined to further expand the data backup. Worker bees data pieces selected program to a repository path routing rules, and cross-slicing at the application layer. Hundreds of thousands of warehouses located in different clusters, enables seamless migration between clusters and cluster dynamic expansion.
Solve the storage problem after expansion, traffic increased gradually exposes performance bottlenecks stand-alone, read and write the code base are concentrated on a single host, it can lead to computing and memory resources tight. By analyzing the sources, a large number of read requests from compiling and publishing system for reading and writing this little scene, the worker bees to achieve a separate read and write mode a master multi-level code base from the write request to the hosts, a read request is based current load balancing to split from the machine. Git primary synchronization using the native operating data from among the maximum extent to ensure data consistency and atomicity operations. At the same time as the slave hot standby data in real time, with off-site cold standby, set up a complete code base data disaster recovery system to ensure data security. Figure 1 is a complete code base back-end storage architecture.
1 separate read and write data fragmentation and
how to manage large libraries has been a problem code management tool, Git is designed to manage text file is the code of the class, but the project will inevitably be dependent libraries and resource files, especially Tencent game class of business, contains a large number of pictures, audio and video files, so that this problem is more prominent in Tencent. Worker bees introduced the open source extension program Git LFS, specialized management of large binary files. As illustrated, the outside by Git repository, Git repository retained in the file store these files only text pointer, this approach can greatly reduce the volume of 2 Git repository itself, speed cloned repository. Currently a single big game Warehouse worker bee management of more than 2.5T, single library ceiling issue is resolved.
2 large files stored
on the overall architecture, the industry's worker bees using the popular micro-services architecture. 3, the proxy service provider to the protocol HTTP, SSH, LFS three independent access link protocol, encapsulates the data service database access, routing service request data for each node of the rear end of the code base address, business service platform according to split function provides, for example, code browsing, code, statistics, code review, code search, etc. are independent of micro-services. In addition, a unified registry and configuration center provides service discovery, service routing, and service configuration abnormal fusing such global functions. All services are designed for micro stateless mode, it can easily scale horizontally. With the ability of the vessel to deploy, ready to adjust the number of instances in order to cope with high concurrency scenarios.
3 micro Services Architecture
Code tool if you do not open up the upstream and downstream development process, to enhance the effectiveness of R & D role is very limited. One of the advantages of worker bees is to enrich the ability to open and support third-party systems integrators access. Webhook push mechanisms to facilitate third-party code libraries to subscribe to submit an event, widely used to automatically trigger post-commit code compiler to build continuous integration system. Commit Check intercept mechanism for engagement into the code before initiating automatic assembly coding standards, defect detection, unit testing, the code checks the quality by providing red, strictly controlling the quality of the combined code. Worker bees also provides a rich API restful meet the standards, improve the private OAuth tokens and authorization mechanisms, to third parties to provide standardized access to safe and effective way to expand the application scenarios worker bees.
Tencent internal, worker bees have universal coverage in the six business groups, serving thousands of lines of business, including micro letter, QQ, including the number of the code base of nearly two hundred thousand daily visits amounted to millions of levels, API number of daily calls million, effectively enhance the company's overall R & D performance. In-house open source collaborative strategic objectives, the worker bees are subtle changes in the company's collaborative approach, the worker bees of the current project has achieved more than half of the internal completely open source, with Issue discussion is also an effective way to communicate cross-team collaboration .
The end of September this year, "Tencent worker bee - Git R & D based engineering platform" project stand out in the selection of China Computer Society, won the 2019 "CCF Science and Technology Award." It is reported that, "CCF Science and Technology Award" granted subject to important discoveries, inventions, original innovation in computer science, engineering or technology, there are certain international influence of the outstanding achievements in related fields. This award is a great recognition of worker bees, worker bees in the future will focus on the degree of code reuse, exploring the integration of research and development experience in many aspects, such as R & D process data metrics for deep plowing in the field of management codes, provide for the company and the industry more value.