On December 03, 2022, the Apache Software Foundation (ASF) incubator passed the incubation graduation vote for the Apache Linkis computing middleware project. On January 18, 2023, the Apache Software Foundation officially announced that Apache Linkis had successfully graduated and became an Apache Top Level Project (TLP).
The screenshot above is from Apache official website
Original link: https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-apache-linkis-as-a-top-level-project
1. About Apache Linkis
The Apache Linkis computing middleware project was open sourced by WeBank's big data platform team in July 2019, and officially donated to the world's top open source community Apache Software Foundation (ASF) in August 2021, becoming an ASF incubation project. Pass the hatch graduation vote on December 03, 2022.
Apache Linkis builds a layer of computing middleware between the upper application and the underlying engine. By using standard interfaces such as REST/JDBC/Shell provided by Linkis, upper-layer applications can easily connect and access lower-level engines such as MySQL/Spark/Hive/Trino/Flink, and at the same time realize cross-upper-layer user resources such as variables, scripts, functions, and resource files Application interoperability, as well as data source management and metadata query services corresponding to data sources are provided through the REST standard interface. As a computing middleware, Linkis provides powerful connectivity, reuse, orchestration, expansion and governance capabilities. The application layer and the engine layer are decoupled through computing middleware, which simplifies the complex network call relationship, reduces the overall complexity, and saves the overall development and maintenance costs.
Since the open source, the total number of users in the open source community group has exceeded 7,600, the sandbox trial companies have exceeded 2,600, and more than 110 companies have received feedback that they have put into production. The data volume supported by the production environment exceeds 400PB, and the number of users of production services exceeds 5,000. , involving finance, telecommunications, manufacturing, Internet and other industries. Many companies have used Linkis to solve computing governance problems such as connectivity, expansion, control, and orchestration of big data platforms.
During the incubation period of more than a year, Apache Linkis has released 7 Apache versions led by community developers, and an average version will be released in about two months; 4 new PPMC members and 13 Committers have been added, from different companies and Team, the number of contributors reached 127. The indicators of the Apache Linkis community are developing well, and the project has matured. Apache Linkis successfully passed the graduation evaluation and voting of the ASF Incubator Project Management Committee, and was approved by the ASF Board of Directors. After graduation, it was promoted to the top project of the Apache Foundation.
2. The core functions of Apache Linkis and Roadmap
By using Linkis, upper-level application tools such as batch systems can use Linkis as a unified gateway or JobServer to submit batch tasks; data analysis tools can use Linkis' interactive task capabilities to obtain logs, result sets, and progress information in real time. Tools integrated with Linkis can be interconnected, and the same defined UDF, data source, material, etc. can be used in different tools.
The upper-level application tools connected to Linkis can have the following core features:
-
Rich underlying computing and storage engine support: Spark, Hive, Flink, Python, Sqoop, OpenLooKeng, Presto, Trino, ElasticSearch, JDBC, SeaTunnel, Shell, etc.;
-
Rich language support: SparkSQL, HiveQL, Python, Shell, Pyspark, Scala, JSON, and JAVA;
-
Powerful computing governance capabilities: able to provide multi-level tag-based task routing, load balancing, multi-tenancy, flow control, resource control and other capabilities;
-
Full-stack computing and storage engine architecture support: able to receive, execute and manage tasks and requests for various computing and storage engines, including offline batch tasks, interactive query tasks, real-time streaming tasks and data lake tasks;
-
Unified context service: Support cross-user, system, computing engine to associate management user and system resource files (JAR, ZIP, Properties, etc.), result sets, parameter variables, functions, UDF, etc., one setting, everywhere automatic reference;
-
Unified materials: Provides system and user-level material management, which can be shared and transferred, and materials can be shared across users and systems;
-
Unified data source management: Provides information addition, deletion, and modification, version control, connection testing, and metadata information query capabilities for Hive, ElasticSearch, Mysql, Kafka, MongoDB, and other types of data sources;
-
Error code capability: Provides error codes and solutions for common errors in tasks, allowing users to self-locate problems;
Graduating from the incubator also proves that the Linkis project and community have further matured, but there is still a lot to do to become a world-class computing middleware project. The Linkis community has compiled a roadmap, and everyone is welcome to participate and contribute together. Some of the significant features are as follows:
3. Apache Linkis Ecology
Linkis, as a computing middleware, also actively promotes integration with projects in multiple open source communities to enrich Linkis' ecology. The figure below shows some open source projects that have been integrated with Linkis, covering data analysis, workflow, data exchange, data quality, machine learning, and commonly used computing and storage engines. The upper application tools submit and manage the tasks of various engines through Linkis, and use Linkis as a development base. It greatly reduces the development workload of upper-layer application tools in computing governance such as connectivity, expansion, and control, so that upper-layer application tools only need to focus on their own functional requirements.
In the future, we will further guide and promote more upper-level tool systems and lower-level engines to integrate with Linkis and enrich Linkis' ecology. More and more projects are welcome to participate in it.
4. Graduation message (sorted alphabetically by surname)
Congratulations to Apache Linkis for graduating! Along the way, I have been fortunate to witness the development and growth of the Linkis community, and its influence in the field of open source big data is also increasing day by day. I hope that Apache Linkis will develop better and better in the future and become a more influential world-class open source project! ——Du Junping, Apache Linkis Champion, Member of Apache Software Foundation, Chairman of TOC of Open Atom Open Source Foundation
The development of the Linkis community is very prosperous. Congratulations to Linkis for successfully graduating from the Apache incubator and becoming a top project. As an excellent computing middleware in the field of big data, Linkis makes it easier for users to access underlying engines such as Spark and Flink through standard interfaces such as REST/JDBC . Graduation is the beginning of a new journey, and I hope Linkis can develop even better! ——Dai Lidong, Apache Linkis Mentor, Apache Software Foundation Member, Apache DolphinScheduler PMC Chair
Thanks to every member of the community for their enthusiasm and contributions! Through the continuous practice of "Apache Way", we have formed a diverse, orderly, and vibrant community that can promote the continuous evolution of the project. Graduation is just the beginning of a new journey. I look forward to working with you to continue to enhance Linkis' capabilities in data platform computing governance such as connectivity, control, orchestration, and reuse, so as to make big data easier. ——Di Shuai, Apache Linkis PMC Chair, Head of Big Data Platform of WeBank
Congratulations to Apache Linkis for graduating from the ASF incubator as an Apache top-level project. As an Apache incubator mentor, I am very happy to see that Apache Linkis can build a healthy and developing community relying on project users and developers, and practice the Apache Way well. I wish Apache Linkis to thrive and create a more prosperous open source big data technology ecosystem together with other Apache big data projects! ——Jiang Ning, Apache Software Foundation Director, Apache ServiceComb PMC Chair
I'm glad to see that the computing middleware Linkis, which started from a staggering start in Wezhong, can grow rapidly through open source, under the guidance of the Apache Way and the joint cultivation of community partners, enter all walks of life, serve thousands of enterprises, and successfully graduate to become Apache top-level project. It is expected that Linkis will continue to thrive and play greater value in the field of big data open source. We also look forward to the participation of more community partners, and the emergence of more excellent open source projects, which will continue to contribute to the development of basic software open source. ——Lu Daohe, Head of Basic Technology Products Department of WeBank
Congratulations to Apache Linkis for adding another strong member (20 projects in total) to the top Apache projects originating from China at the end of 2022. I wish the journey of Apache Linkis computing governance goes smoothly, and the inclusive community continues to expand!
——Liu Tiandong, Member of Apache Software Foundation, Co-Founder of Open Source Society
Congratulations to Linkis for successfully graduating from the Apache incubator as an Apache top-level project. As an excellent open source and donation project led by WeBank, the computing middleware Apache Linkis has been continuously polished in a financial-level production environment and jointly built and improved by contributors from various industries in the community. It has become a powerful tool to solve computing governance problems on big data platforms. In the future, WeBank will continue to support the construction of open source communities and help my country contribute to the development of global basic software. ——Ma Zhitao, Vice President and Chief Information Officer of WeBank
As a heavy user of Apache Linkis, I am very happy to see it graduate from Apache, and I would like to express my sincere blessings here. We started to pay attention to and try to use it in business from 2020, and have migrated most of the underlying business to Linkis for management. In the process, we have received strong support from our community partners and also contributed our own strength. Graduation is not the end, but a new beginning, and there is still a lot of work waiting for us to do. Hope to work with community partners to make Apache Linkis stronger and more perfect in the future. ——Nie Lei, Apache Linkis Contributor, Big Data Architect of Ideal Automobile
Apache Linkis is a big data project that was open sourced by WeBank and contributed to the Apache Software Foundation in 2019. Linkis has attracted widespread attention in the industry since it was open sourced. What's even more commendable is that the developer community formed around the Linkis project is very active. Currently, there are 23 members of the project management committee, 33 code committers and more than 127 contributors. They come from different regions and organizations and spontaneously participate in the project. Develop new features, answer user questions, fix bugs, release versions, etc., showing good diversity and sustainability. In December 2022, the Linkis project passed the Apache incubator's project maturity assessment and voting, and graduated to become another top Apache project. For Linkis, graduation is not an end, but a milestone. In 2023, I hope that the Linkis community can continue to uphold the concept of The Apache Way, absorb more users and developers with the spirit of Community Over Code, and continue to develop and expand open source software projects so that they can ultimately benefit more businesses and individual users .
——Shi Shaofeng, Apache Linkis Mentor, Apache Software Foundation Member, Apache Kylin PMC Chair
It is great to see that the Linkis project donated by WeBank to the Apache Open Source Software Foundation has graduated as a top-level project of the Apache Foundation. This proves that the project sponsors fully understand the values of the Apache Foundation, Community Over Code, and follow the open, transparent, and meritocratic rules of the Apache community for long-term operation. The maturity of the community has been unanimously recognized by the Incubation Committee of the Apache Software Foundation. Congratulations to the Linkis project and team for providing a good project worthy of long-term trust for the big data technology ecosystem. The decoupling of the application layer and the engine layer through computing middleware makes the writing of upper-level big data applications easier and faster. ——Tan Zhongyi, Member of Apache Software Foundation, Deputy Secretary of China Open Source Promotion Alliance, Vice Chairman of TOC of Open Atom Open Source Foundation
Congratulations to Linkis for graduating from the Apache Incubator! Liniks is a computing middleware for Hadoop and related ecosystem "database". It is a very interesting and valuable project. Here, I sincerely wish the Apache LInkis project and the community to develop better and better! ——Wang Hualei, Apache Linkis Contributor, Deputy Chief Engineer of the Data Management Department of Postal Savings Bank of China
As a user and developer of Linkis, I am very honored to witness Linkis from open source, to Apache incubation, to final graduation. It is really not easy, and I have also witnessed the rise of China's open source power. With the contributions of more and more small partners, Linkis has become more and more stable. We look forward to more small partners joining the loving Linkis community and building a better community together. ——Wang Xiaogang, Apache Linkis PMC Member, Tianyi Cloud Big Data Development Engineer
As the Mentor of the project, I am very happy to see that Apache Linkis has successfully graduated as a top-level Apache project. The community of Apache Linkis is very active. During the incubation period, it has always maintained a release rhythm of once every two months. This is very rare among all incubation projects, which shows that the participants of the project are very active and dedicated. I wish the Apache Linkis project better and better in the future! ——Zhang Duo, Apache Linkis Mentor, Apache Software Foundation Member, Apache HBase PMC Chair
Congratulations to Apache Linkis for graduating from Apache! We have started to pay attention to and try to apply it to some businesses in 2021. Currently, some businesses are running on Linkis, and we are now working on the migration of other businesses. Thank you so much for your patience and help in the community! Becoming a top project is not the end, but a beginning. I sincerely hope that Linkis will develop better in the future and help more companies build big data platforms. ——Zhang Huajin, Apache Linkis PMC Member, Senior Demand Analyst, Bank of Guangzhou Credit Card Center
As a loyal user and developer of Apache Linkis first put into the production environment, I am very happy to witness the development and growth of Linkis step by step, and I am honored to contribute a little bit to the community. I wish the Apache Linkis community will develop better and better, and contribute more value to the big data platform. I hope everyone will participate and build Linkis even better! ——Zhu Hui, Apache Linkis PMC Member, head of BOSS direct employment data center
Congratulations to Linkis for graduating from the Apache incubator and becoming a top project after 16 months of incubation. During the incubation process, the Linkis community has always followed the Apache Way, adopted an open, transparent, and inclusive governance system, and established a very prosperous and diverse open source community. Congratulations to all the contributors and participants in the Linkis community, and wish the Linkis open source project can develop better and better! ——Zhong Yanqing, Head of Open Source Management Office of WeBank
5. Looking forward to your joining
Graduating from the Apache incubator is just the beginning, the first step towards becoming a truly top open source project. The Linkis community will continue to practice Apache's open source culture and create a more collaborative, open and diverse community. We welcome more organizations and individuals to participate in the construction of Apache Linkis to promote the sustainable and healthy development of the Linkis community.
6. Apache Linkis official link
Official website homepage:
https://linkis.apache.org/
Warehouse Address:
https://github.com/apache/linkis
Subscribe to the Linkis mailing list: [email protected] (after sending any content to the mailbox, subscribe to the mailing list according to the reply)