Cooperation, participation, making open source more accessible | Amazon's open source culture

JAX is an increasingly popular library that supports composable function transformations of native Python or NumPy functions for high-performance numerical computing and machine learning research. JAX provides the ability to write NumPy programs that can be automatically differentiated and accelerated using GPUs/TPUs, resulting in a more flexible framework to support modern deep learning architectures. In these two articles, we discussed some topics about JAX, and we hope it will be helpful for you to use JAX as a framework for deep learning research.

The Amazon cloud technology developer community provides developers with global development technology resources. There are technical documents, development cases, technical columns, training videos, activities and competitions, etc. Help Chinese developers connect with the world's most cutting-edge technologies, ideas, and projects, and recommend outstanding Chinese developers or technologies to the global cloud community. If you haven't paid attention/favorite yet, please don't rush over when you see this, click here to make it your technical treasure house!

You can also see that with the rapid development of the mobile Internet and the Internet of Things, various cloud computing technologies are widely used by both individual developers and large-scale enterprise services. Among the technologies supporting these cloud computing facilities, open source products are widely distributed. As a leader in the industry, Amazon Cloud Technology not only widely uses various open source technologies to support complex and powerful cloud computing services, but also participates in the development and promotion of open source projects, actively promoting the development of open source technologies and ecological construction.

Amazon Cloud Technology has always maintained a positive and open attitude in the open source community:

Wider cooperation : We promote the development of open source technology by establishing close partnerships with open source users, partners, and communities. In these cooperations, we make the cake of open source bigger through direct code contribution and technology promotion, so that all parties involved in open source can fully develop;
Higher participation : Through cloud computing, we benefit from open source on the one hand, and feed back open source on the other hand. We firmly believe that the continuous vigorous development of the open source ecology is conducive to the formation of a virtuous circle and can promote the prosperity of the entire open source community and ecology;
Make open source easy to use : We integrate open source technology with automated and fully managed cloud computing services to reduce the cost of using open source technology and promote the implementation and development of open source technology in production. In this process, more users can use open source technology at a lower cost, and promote the digital transformation of more industries.

Amazon's leadership principles are the core of Amazon's culture. It is like Amazon's DNA integrated into every important decision-making, deeply affecting every Amazon person, every Amazon customer, partner and every Amazon cloud Builders of technology. At the same time, Amazon's leadership principles have a profound impact on the way Amazon interacts with open source. Watch this part of the video presentation on the Build On Cloud video account:

Customer first

The first and very important one of Amazon’s leadership principles is: the customer comes first.

“There are many advantages to a customer-centric approach, but the biggest advantage is that even when customers say they are happy and agree that our business is great, there is always a sense of dissatisfaction.”

“Sometimes they don’t even realize it themselves, customers always want a better experience and service, and your desire to please them will drive you to create on their behalf.”

—— Jeff Bezos

Customer first, is always customer-centric. When it comes to what products and services to deliver, Amazon Cloud Technology will first listen to the needs of customers and developers. We will not build technology for the sake of technology, nor will we rush to innovate for the sake of innovation. As a result, 90% of Amazon's cloud technology product roadmap is driven by direct feedback from customers and developers.

The same goes for open source. Most developers are also contributors and promoters of open source technologies. They told us that they like the inclusive and open environment and creativity of open source, and they also hope to obtain more extensive and mature products and services from the cloud. They see deep integration between services leading to significantly greater agility and manageability.

Developers say they like to decide what to run and how to run their applications built with open source technologies according to the criteria they choose. Sometimes, they want to self-manage these open source workloads, but in some business scenarios, they will use the tools and services provided by cloud providers to run and maintain their open source applications.
For Amazon cloud technology, we need to deeply understand the specific problems and needs of developers when developing and running open source software, and help them solve these problems.
Amazon cloud technology provides a variety of Apache Airflow service methods according to different application scenarios: Apache Airflow is an open source project launched by Airbnb in 2014. Its purpose is to manage increasingly complex data management tools, scripts and analysis tools, and provide a framework Scenarios for batch workflows. From a functional point of view, this is a scalable distributed workflow scheduling system that allows workflows to be modeled as directed acyclic graphs (DAGs), which simplifies the creation and orchestration of each processing step in the data pipeline and monitoring. Project address: airflow.apache.org

Amazon cloud technology provides self-management

Compared with deploying, running, and managing Apache Airflow in their own data centers through fully open-source Kubernetes, developers can use public clouds to deploy, run, and manage Airflow on fully managed container services, such as Amazon ECS and Amazon EKS. It saves manpower and time for infrastructure operation and maintenance, configuration and reservation of computing resources, and cluster resource management to a large extent, and also obtains more tools, service choices, and more flexibility. Deeply integrated cloud hosting services will also help developers complete complex and tedious tasks such as automated IT operation and maintenance management, elastic expansion, and high availability. Let them focus more on the development and operation of Airflow itself.

See Running Airflow on Amazon Fargate for details :

However, the self-management approach still faces the following challenges:

Deployment configuration is cumbersome . Configuration management of open source software is often a completely manual process. For many developers, large-scale deployments require custom development tools for their own environment. At the same time, it is not easy to choose the most suitable configuration for the various options in the deployment configuration process;
It is difficult to expand . Although developers can complete the automatic expansion of virtual machines or container cluster expansion on the cloud, these two configuration management tasks themselves have a certain degree of complexity. Developers need a certain amount of technology accumulation, and corresponding project costs will be incurred in production;
High security risk . In terms of security, supporting role-based authentication and authorization usually involves a process. Developers must authenticate in one place and then go into the Apache Airflow user interface to authorize the specific role (such as administrator or viewer) to which that particular person belongs. In order to save configuration time, many developers often authorize all developers to become administrators. But from a security point of view, this is a very dangerous thing, which can easily cause project management confusion and human security risks;
Version upgrades are complex . There are hundreds of Python libraries in Apache Airflow, and among these libraries and other dependencies, it is almost impossible to track and pinpoint which versions are stable, which versions are required, and which versions are security risks. The pain of upgrading Apache Airflow often occurs in the rollback process after the upgrade fails.

Amazon cloud technology provides a fully managed service - MWAA

In order to solve the above pain points and challenges in the development process for developers, in December 2020, Amazon Cloud Technology launched the fully managed service of Managed Workflows for Apache Airflow (MWAA) for Apache Airflow.

Managed Workflows for Apache Airflow（MWAA）

Through the Amazon cloud technology management console, CLI tools, Amazon CloudFormation or Amazon SDK, developers can complete the deployment of the basic environment in a few minutes, start account creation, and deploy the "directed acyclic graph" to their own Apache Airflow environment without relying on development resources or pre-built infrastructure. This makes it easier for developers to set up and operate end-to-end data pipelines on a large scale in the cloud; this service also provides an upstream version of Apache Airflow, which effectively solves the problem of self-management such as configuration, expansion, and upgrade patches , greatly improving production efficiency. And MWAA can be integrated with other services in the developer's Amazon cloud technology service account to provide more automation.
With a fully managed MWAA, developer data is more secure - because Apache Airflow can be run with built-in security. First, workloads will run in a secure environment that is isolated from each other provided by Amazon VPC. Secondly, the data will be automatically encrypted using Amazon Key Management Service (KMS). For the user management interface of Apache Airflow, developers can use Amazon Identity and Access Management (IAM) to control role-based authentication and authorization, and provide single sign-on access.
With the help of MWAA, developers can avoid the tedious and complex deployment, configuration, expansion, security and other work required for large-scale operation of open source Apache Airflow, save the cost of learning more service tools in development, and reduce the corresponding engineering work in production. overhead.

Amazon cloud technology provides a richer ecological environment

The rich ecology of Amazon cloud technology enables Apache Airflow to get more help in deployment and operation. Developers also have the option to use Apache Airflow through some Amazon cloud technology partners. A rich ecology can get more help in deployment and operation. We can see this from the various services provided by the Amazon cloud technology marketplace: AWS Marketplace: Search Results

In addition, in order to make the testing and development of the Apache Airflow workflow more flexible and agile, developers hope to use local resources and familiar IDEs to test and develop the Apache Airflow workflow. To help developers achieve this goal, Amazon Web Technologies contributed a related open source project, aws-mwaa-local-runner . It helps developers develop and test their Apache Airflow workflows locally using local resources before committing to the live environment.

When a developer wants to find and use the source code in a specific project, it takes a process to find the source code and use it. When you dig into the details, you can see that operating and benefiting from open source technologies is complex and tedious work. Of course, this is why developers like to have multiple options for using open source technologies. For Amazon, customer first means always focusing on customers and developers. In the field of open source, Amazon always believes that customer first is to let developers open source in their own way, so no matter which way our developers choose , we will stand with them in the end.

In the next sharing, we will explain to you another leadership principle that guides Amazon's open source culture - the highest standard. This is an important principle for us to create self-managed, fully managed services and ensure stability and reliability.

Welcome to continue to pay attention to the Build On Cloud WeChat official account to learn more about technology sharing and cloud development trends for developers!

Past recommendation

Author Zheng Yubin

Senior developer evangelist of Amazon Cloud Technology, with 20 years of experience in the ICT industry and digital transformation practice, focusing on the field of cloud native and cloud security technology of Amazon Cloud Technology. With 18 years of experience as an architect, he is committed to providing consulting and technical implementation of data center construction and software-defined data center solutions for finance, education, manufacturing, and Fortune 500 corporate users.

Article source: https://dev.amazoncloud.cn/column/article/63e33987e5e05b6ff897ca10?sc_medium=regulartraffic&sc_campaign=crossplatform&sc_channel=CSDN