Open source is the foundation of the software world. Bytedance values the long-term value of participating in open source. Its attitude towards open source has always been open and encouraging, and it is willing to work with global partners to achieve ecological prosperity.
This article is compiled from the keynote speech given by Zhang Xin, head of governance and operations of the ByteDance Open Source Committee and vice president of Volcano Engine, at the OSPO Summit.
The Open Source History of ByteDance
The open source process of ByteDance has generally gone through three stages: from use, to participation, to active open source. In the early days of Byte’s business development, we widely adopted open source technology to quickly build a technology platform, which promoted the development of ByteDance’s businesses including Douyin and Toutiao. With the deepening of open source use, we optimize and improve open source technology according to our own business scenarios, and feed back to the community, and gradually begin to participate in open source contributions. After the accumulation of contributions, Byte students began to try to systematically take out some of their own technological breakthroughs to form a complete project and give back to the open source community, which is the active open source stage.
By 2022, ByteDance has open sourced more than 50 projects. With the gradual increase of active open source projects, we found that everyone has open sourced many technologies and tools in various technical fields, but the company still lacks a top-level strategy for open source. Think and design plans. For open source projects, there is also a lack of good operations, and there is very little experience in how to operate an open source community and how to cultivate developer relationships. Another bottleneck encountered is how to ensure open source compliance and not touch the red line.
In this context, the company decided to establish a middle-end organization such as OSPO, mainly for the following purposes:
-
-
Formulate company-level open source strategy: clarify the value, goals and strategies of open source, and ensure systematic and targeted open source work.
Clarify the review criteria for open source projects: Screen and focus on high-value key projects to ensure the resource support and effect output of key open source projects.
Strengthen open source management norms: optimize the approval process to ensure the legality, compliance and reasonableness of various open source practices such as use/contribution/self-developed open source.
Do a good job in internal and external open source evangelism: internally improve open source awareness and education, externally demonstrate open source achievements, and build community influence.
-
Open source is positioned in the company as a long-term cause that can promote our own technological progress and at the same time promote the development of industry technology. ByteDance OSPO has also become a middle office department in the company. Its organizational structure is not subordinate to a certain business unit. It is led by CEO Liang Rubo, Yang Zhenyuan, and Hong Dingkun, and is divided into two groups: strategic review and governance operation. There are also many collaborative teams to support together.
We hope that Byte Open Source can adhere to the long-term principle, do a good job of few but fine quality projects, and adhere to a few principles when implementing them:
-
-
For the value consideration of open source results, replace short-term KPIs with more long-term North Star indicators
Building boutique projects is better than broad coverage
User value is better than commercial realization, focusing on solving real problems of users
Security and Compliance Are the Bottom Line
-
Value thinking and practice of open source
On the whole, an open source project can obtain certain "commercial value" after passing the test of market influence and acceptance. We believe that the "commercial value" here is not short-term commercial realization, but includes improving technological influence through open source, which can better enrich the developer ecosystem, support business development, and attract talents. Through open source and community participation, we can also better promote internal technological innovation and avoid closed doors and industry disconnection. Only by opening up can we compete with industry experts, accept the test of customers' real scenarios, and accelerate technological evolution.
In the stage of using open source
Open source software and technologies can help companies solve infrastructure problems and accelerate core business iterations; access to open source standards can lower recruitment barriers. However, in the process, there will also be legal, technical, security and other risks. At the same time, problems such as blindly pulling branches and being out of touch with the upstream will also be encountered.
In our practice, the first thing that needs to be solved is that there are risks in security, operation and maintenance, and compliance in the open source software supply chain. Doing a good job in open source compliance governance is the most basic and critical work in the use of open source. In order to solve this problem, we have gradually started the open source compliance governance work three years ago, mainly from the three aspects of "infrastructure construction", "compliance governance" and "training publicity" .
Infrastructure construction includes drafting company-level open source policies at the system and process level, publishing governance guidance documents such as open source license compliance guidelines, and creating an open source compliance scanning platform for "one person, one account".
In terms of compliance governance, we carried out special governance on open source compliance for each business line, carried out full code warehouse scanning and blue army directional mining, etc., and also organized training courses related to open source compliance.
The implementation of open source compliance involves a lot of details, and the composition of the team also includes legal affairs, security, tools, compliance, QA, R&D, and product functions. At present, the team has introduced the industry's mature OpenChain open source compliance international standards, and formed supporting policies, systems, processes, and internal and external publicity mechanisms to ensure efficient and compliant use, contribution, and active open source, and then guarantee from the perspective of the open source software supply chain The company's credibility in the open source community.
In the stage of contributing to open source
Participating in open source community contributions can improve the quality of open source software, help form a virtuous circle, increase the sense of identity of enterprise employees, attract or retain talents, and also allow internal teams to contact excellent external developers to improve the team's technical capabilities. Open source is the cradle of innovation. Contributing to open source can help enterprises plan the future development of business and technology more strategically.
A common misunderstanding at this stage is that contributing to open source means contributing code. In fact, open source contributions are not limited to code, including technical documents, conference sharing, organizing/participating in community activities, etc. At the same time, conflicts caused by lack of misunderstanding of community codes of conduct, norms, etc. will also be encountered.
The ByteDance Open Source Committee is committed to promoting the value of upstream contributions within the enterprise, and is exploring ways to better motivate developers and cultivate the habit of upstream contributions.
Take the Systems Technology and Engineering Team (STE Team) as an example:
The STE team has been committed to research and exploration in the field of system technology. At the same time, it actively participates in the construction of open source communities and technical contributions, conducts community collaboration and technological innovation around the kernel, virtualization, operating system and other fields, and feeds back years of technical optimization experience to the community. Build a good open source ecosystem of basic software with developers. Including submitting 450+patches to the Linux kernel community and 270+patches to the Open BMC community. Many students have also become maintainers in the open source community.
In addition, the team also contributes or maintains multiple independent modules to the community, and the proposed HVO solution has been integrated into the community, solving problems such as Linux kernel memory management redundancy, and has been recognized by the industry.
In 2020, the STE kernel team officially open-sourced VDUSE, a next-generation high-performance device virtualization framework for cloud-native scenarios, to the Linux kernel community, providing a unified I/O virtualization layer for containers and virtual machines. This solution has also been deployed on a large scale in Byte's cloud-native scenarios. After a year, VDUSE was officially incorporated in Linux version 5.15. Currently relying on the Linux kernel, the team is continuing to explore technical directions such as virtualization, cloud native, and eBPF.
Making basic software such as operating systems is not a temporary enthusiasm, but requires long-term investment. The big waves wash away the sand and the sand is exhausted, and when the sand is exhausted, real gold can be seen.
—— Zhang Yu, head of ByteDance STE team, member of TOC strategic review group of Open Source Committee
In the active open source stage
Open source can help promote technical standards, and enterprises can build technical influence through contributions to core communities, open source community activities, and open source foundation contributions. In addition, the needs of the open source community can also help introduce more users to polish products, derive more applications and services, build a developer ecosystem, and help core business development. The misunderstanding that should be avoided at this stage is that open source essentially ensures "openness", "use", "modification", and "distribution", and it is also necessary to avoid open source technology-only theory.
In terms of active open source, after years of accumulation, ByteDance has developed many projects in the fields of AI, big data, architecture, security, front-end and audio and video, such as CloudWeGo, a middleware collection of enterprise-level cloud-native micro-service architecture. , ByConity, a data warehouse engine that separates storage and computing, lightseq, a high-performance transformer acceleration engine, BytePS, a distributed deep learning communication framework, KubeWharf, a collection of cloud-native projects, etc. This year, we will open source very valuable projects in AI/ML compilers, privacy computing and other fields, and look forward to your participation and support.
Finally, I would like to quote Bytedance's mission of "inspiring creation and enriching life", which is actually applicable in the field of open source. We hope that through open source and OSPO, we can stimulate more technological innovation, as well as the potential of the community and the industry, so that more people and organizations can enjoy the creation process and value, and better bring together individual innovative ideas through open source projects , thereby enhancing the creativity of communities and organizations.
If Douyin and Toutiao let hundreds of millions of users see a more colorful world, open source is to help more people better self-improvement and realization through collaboration with more people, enriching the breadth of life, whether it is an individual or an organization and depth.
Hope to have more in-depth cooperation with all partners in the industry.
About ByteDance Open Source Committee (OSPO)
The ByteDance Open Source Committee (OSPO) is the coordinator at the enterprise level, aiming to formulate strategies to help ByteDance open source succeed. OSPO will improve the open source software management system, promote open source compliance governance and open source operation, and accelerate the vitality and growth of the open source community.