On the evening of August 17, 2023, in a quiet conference room on the 23rd floor of Wangjing Pohang Center. The room was filled with peace, just like the emotions inside now. I know that we must write down some words to record our feelings and thoughts about the ODC open source process. For us, ODC's open source is not only a project milestone, but also a journey full of thinking and discussion.
ODC (OceanBase Developer Center) open source is a small representative of many commercial product open source. In today's environment, the open source tool products of large manufacturers are often questioned. I also hope that through this article, I can share with you some in-depth thoughts on the significance of ODC open source. I also welcome your suggestions and discussions.
About a month ago, we decided that August 18, 2023, would be the day when ODC 4.2.0 version would be released and open source. ODC 4.2.0 is a very radical version. ODC's risk management and control core domain model and page information architecture have been redesigned, and the functions of almost all modules have been enhanced. According to past experience, if the Release date of a large-scale new version is determined, it will definitely be delayed. In order to avoid delays, we set the deadline for August 16th, and it turned out to be very effective. All work related to today’s version release was successfully completed, including code security scanning, compliance scanning, external disclosure, and code synchronization to github and docker images. Pushing to docker hub, everything is silky smooth. The silky smoothness on the eve of the release is the result of months of hard work by the team, not only the ODC project team, but also many students who participated in it.
There are some issues that the team has been thinking about in the past year, such as the value, necessity, and competitiveness of ODC open source. The answers to the questions have always existed, but they are also gradually evolving. Today I met Teacher Yang and had another discussion. Teacher Yang did not answer me directly, but he talked about the logic of why OceanBase should be open source. In fact, the logic is very simple. In the current competitive landscape of the database market, the commercialization of a new database product must be achieved. Success, if you don't take the open source route, the possibility of success is almost non-existent.
Since last year, more and more ODC customers have asked for support for multiple data sources. ODC's idea is simple. Since there are so many customers with needs, then arrange it. But it's not that simple. As a development tool developed by the database manufacturer, ODC was originally positioned to provide OceanBase developers with a graphical development tool that supports necessary functions in the development process such as SQL development, data import and export, and PL execution and debugging. ODC to OceanBase is similar to Oracle's self-developed SQL Developer and SQL Server's self-developed SQL Management Studio. Why should database manufacturers adapt to other database products? Investment should also be made in making tools for migrating from other database products to their own database products more useful. In the first half of last year, we thought that MySQL would be supported soon, but by the second half of the year it still had not been launched.
After a while, some customers expressed the hope of getting the source code, that is, targeted open code. This requirement is actually a simple and difficult thing for the R&D team. To make the code oriented to open source, the transformation cost is very high. Internal dependent components need to be stripped off, and all source code must undergo strict data security and compliance scanning. Then targeted open source is a one-time deal. Each orientation requires a transformation. The code may also fork while being open source, which will bring continuous costs to the maintenance of later versions. Later we learned that in fact, the OceanBase kernel had encountered similar problems, so the solution was very clear. Why not just open source it? If ODC is open source, then it will be very reasonable to support multiple data sources. For example, the demand for supporting MySQL is indeed very strong, and customers can also participate in product co-construction.
So we started to discuss ODC open source, what is its value, what is its goal, and how to measure it? From the perspective of OceanBase as a whole, whether ODC is open source or not has nothing to do with supporting current OceanBase customers. It just adds a component to the OceanBase open source community and attracts some external participants through the community operation area. It is true that the OceanBase core has 5 million lines of C++ code. For interested participants, it will take a relatively long time to invest before they can start participating in community contributions. ODC's technical architecture is based on WEB. From a programming language perspective, Java and JS are easier to use than C++. However, the reason for open source ODC for this benefit is still not sufficient. An open source senior told me that for the same scale of functional development, the investment is expected to be 2 to 3 times higher than if it were not open source, and the input-output ratio is too low.
One aspect that ODC is very sure of is that the value of open source to ODC lies in bringing users closer to the product development team. Through wider user access and communication, more specific product value needs can be screened out, and it can also help ease the core functions. Polish with sex. The experience of making tool products in the past few years tells us that users know better than me what the problems in the current work scenario are, and how several roles collaborate to complete a thing. Therefore, the current ODC R&D process will invite users to participate from the PRD and design draft stages, discuss it with the product R&D team, and then formulate it into the released product. Compared with the previous one-way information transmission, the requirements are from customers to frontline to products and then to the research and development process. The heavy user participation method can realize the heavyweight functions and can be fully used in the customer environment after the first release. After one or two The next few iterations can make it smoother.
Last year, we communicated with some ISVs. Everyone expressed interest in ODC open source but no one was willing to participate in co-building the code. The root cause of the interest was actually the open API and basic component library. This year we know that it is actually The target audience is not right. An open source project can attract collaborators only if participating in the project can bring value to the collaborators. Therefore, it is very important that the project itself is attractive enough and can complement the current business demands of the partners. Well, the time has actually come this year. On the one hand, ODC's product maturity has been greatly improved compared to last year. On the other hand, the concept of database collaborative development has also been popularized, and there are actually few choices for products that are open and easy to use in the industry. many.
At the beginning of this year, the ODC open source project was fully discussed in the OceanBase Technology Department and the Ant Group Open Source Committee. We believe that the necessity of ODC open source is to make it easier for ISVs and customers to integrate OceanBase into their own systems, thereby increasing the activity of the OceanBase ecosystem. , Reduce OceanBase delivery support costs. This is why ODC is open sourced this time. It not only opens up the code of ODC, but also opens up the basic components accumulated during the ODC development process. For example, db-browser encapsulates access to database dictionary views, and ob-sql-parser encapsulates access to database dictionary views. SQL parsing is encapsulated. This may be the most comprehensive SQL parsing library in the Java open source ecosystem. More components will be gradually opened in the future. Making ODC products open source is an important supplement to the OceanBase open source community. As a developer product, open source code is a high-quality supplement to OceanBase documentation.
On February 17, 2023, the ODC open source approval process was officially submitted. After half a year, the day finally came to meet everyone. There is a joke about why big companies don’t open source. It’s not that they don’t want to open source, but that the code is too poorly written to be open source. We are an optimistic and solid team. Once the code is developed, we are not afraid of being criticized. Being criticized is an important way to grow rapidly. It is a little pity that the submission history of the ODC code repository is the second neatest and clearest among all the code repositories in the world that I have seen. Because the historical commits contain some sensitive data, there is no way to bring it to the open source this time.
There are many reasons why the process of transforming a software into open source will last half a year. The cost mainly includes three parts. The first is security compliance transformation. It is necessary to ensure that every line of code is compliant, and before release, it needs to be confirmed that it does not contain any sensitive data to verify that the product itself is safe enough. The second is to peel off the dependencies within the company, including the dependency peeling off of the internal basic class libraries and the internal dependencies of the CI/CD infrastructure. The internal basic class libraries need to be changed to open source dependencies or the previous internal dependent libraries should also be made public. CI/CD and R&D collaboration processes need to implement alternatives on Github. The third is the improvement of products and codes. In order for interested developers to participate, we have adjusted the product technical architecture, reduced the cost of functional expansion, improved the readability of the code, and added a large number of documents.
With so much cost to pay, the ODC team still chooses to firmly follow the open source community route. The most fundamental reason is that I think this matter is beneficial to users and products. In the long term, we have seen that database collaborative development can effectively solve the new challenges brought by data growth and is universal. Issues such as ensuring stable database operation, data security and compliance, and collaborative efficiency have not yet been completely solved from the development stage. , the scale of the problems that need to be solved in this field is still very large, and it cannot be solved by one manufacturer, so ODC open source is an inevitable choice.
Regarding the product name, you may ask why it is strange to use OceanBase Developer Center with other databases. We felt the same way and even launched two rounds of large-scale voting to change the name. But because I haven’t found the destined good name yet, if you have any good suggestions, please submit a Github issue to us.
Regarding the selection of open source protocols, ODC Open Source has chosen the Apache 2.0 protocol, which is open enough. If it can complement your product matrix, we encourage you to build your own complete solutions based on ODC open source projects to provide to customers. For OceanBase, this actually helps realize the most ideal enterprise-level software sales model "distribution". From the perspective of database vendors, if it can help more partner vendors obtain commercial benefits, the issues of sales and delivery costs can be compared. Complete solution.
ODC's open source journey has just begun. If you are interested in this open source project and want to learn more details, we welcome participating in the ODC community in any form. Complaints, questions, suggestions, Issues, Discussions, Pull Requests, and Stars are all welcome!
We are ready, are you interested in participating, and let’s build enterprise-level database collaborative development tools together, By Developer, For Developer!
See you at github.com/oceanbase/odc ！