OpenPitrix is an open source application management system cloudy

background

Cloud computing has been used in the vast majority of enterprises today, with well-known cloud service vendor  RightScale recent survey shows that a growing number of manufacturers to use management cloudy. There are too many reasons for customers to choose cloudy management, and the biggest reason than using a single supplier, can lead to being locked. Therefore, how to manage cloudy environment, and automation in a cloudy environment, many companies are becoming just need, and in this one, the management application is particularly important. Further, challenging application is to create a one-stop management platform, to manage the different types of applications, including traditional applications (or application called monomers, or a traditional master-slave, slice, peer- to-peer architecture of distributed enterprise applications), micro-service applications, as well as the recent rapid development of Serverless application, OpenPitrix to solve these problems and students. Sentence described OpenPitrix:

OpenPitrix is ​​an open source project, used in the cloudy environment package, deploy and manage different types of applications, including traditional applications, micro-service applications, and Serverless applications, among which cloud platform includes AWS, Azure, Kubernetes, QingCloud, OpenStack, VMWare Wait.

Micro service, known as micro-services architecture, which is the inevitable trend of programming, business application to create a new main mode selected when. In addition, the open source project Kubernetes has become the de facto orchestration platform leader , it has a unique advantage in the application of automated deployment, scalability, and management of the container. However, there are still plenty of traditional legacy applications users want to move into in the case without changing its architecture to the cloud platform, but for many users, the use of micro-service architecture, or Serverless architecture is quite a long way off, so, we need to help these users to move their legacy applications to the cloud computing platform, which is OpenPitrix very important function.

In the March 27, 2017, QingCloud release  the AppCenter , aims to set up a friendship bridge between traditional enterprise application developers and users cloud platform, the biggest highlight of this platform is that it allows developers to learn at a very low cost It can be a traditional QingCloud applications to run, and having all the features of the cloud, such as agility, scalability, reliability, and monitoring. Typically, a developer need only spend a few hours to understand the whole workflow, and then spend a week or two (depending on the specific application of this complexity) migrating applications to the cloud platform. After the platform has been popular on-line user's favor and praise, but some users have made more demands, they want to deploy it to manage their internal cloudy environment. In order to meet the needs of users, QingCloud extend it, that manage multiple types of applications under cloudy circumstances, and the use of open-source approach to the sound development of the project.

As the saying goes: "easier said than done", although OpenPitrix original team computing application development in the cloud is rich enough experience, and successfully developed a stable business products: AppCenter, you know, waiting in front of the still have a lot of difficulties to overcome. OpenPitrix  from the start to open the way to carry out, and in August 2017 created the organization and project on GitHub, until 24 February 2018 only wrote the first line of the function code, in the meantime, the team All members are thinking about each critical point in the system, the details of these discussions are available on GitHub open access .

The above is introduced OpenPitrix ins and outs of the project, the next will explain some of the detailed functional and design details.

The main function

OpenPitrix achieve the desired functionality include the following:

• Support for multiple cloud platforms such as AWS, Azure, Kubernetes, QingCloud, OpenStack, VMWare and so on;
• Supported cloud platform is highly extensible and pluggable;
• Support for multiple application types: traditional applications, micro-service applications, Serverless application;
• Support of the application is also highly scalable, which means that no matter which type of new applications arise in the future, OpenPitrix platform can support it by adding the appropriate plug-ins;
• Warehousing applications are configurable, which means driven by the OpenPitrix shops, its application can be used are traded;
• Visibility application library is configurable, including public, private, or only allow a specific set of users can access by OpenPitrix driven market, each vendor will be able to operate part of his / her own application store.

Examples of user scenarios

OpenPitrix typical user scenarios are:

• An enterprise system is the use of a cloudy (including hybrid cloud), to achieve a one-stop application management platform to deploy and manage applications;
• Cloud Platform tubes (CMP) which may be considered as one of the components OpenPitrix to implement management application cloudy environment;
• Kubernetes can be used as an application management systems. OpenPitrix and Helm has a fundamentally different, although OpenPitrix bottom with the Helm to deploy Kubernetes application, but OpenPitrix focus on the full life cycle management applications, such as in the enterprise, usually in accordance with the application of state to classify, such as development, testing, preview, production; and even some organizations will be categorized by department, which is Helm does not have.

Architecture Overview

The fundamental idea OpenPitrix design is the decoupling of applications and application run-time environment (instead of using the run-time environment where cloud platform, the same below), as shown below. Applications to run in the environment which, in addition to matching provider information, but also need to match the application warehouse where a selector (selector) and runtime environment label (label), that is, when an end user selects one from the store a specific application, and then try to deploy it, the system will automatically select the runtime environment. If you have multiple runtime environments can run this application, then the system will pop up the corresponding dialog box to let the user choose, more design details, please refer  OpenPitrix design documentation .

Storage Subsystem applications

As shown in the above overview of the architecture, storage subsystem is divided into three components: storage management (Repo Manager), repository index (Repo Indexer), warehousing (App Repo); wherein the storage management including warehouses (Repo Service) and its backend database (Repo DB), the architecture as shown in FIG. Information required to configure to create applications such as warehouse users: URL, credentials, whether visible, usually a system administrator or developer to create warehouse platform. Daemon warehouse index periodically scanning the new warehouse, any updates to the existing warehouse, and updates to the database application management service. We can see, warehouse storage is independent of the specific OpenPitrix platform, so the warehouse storage applications can be shared with any OpenPitrix-based application management platform. For more details, please refer to the design  OpenPitrix warehouse subsystem design .

By the application warehouse organization, it can be used as separate assets sold by the store-based OpenPitrix.

How to migrate legacy applications to cloudy environment

First of all make it clear that, OpenPitrix not particular to distinguish the difference between cloudy and hybrid cloud, in this context, in the absence of special circumstances prompt, cloudy means that multiple cloud environments, whether they are public or private clouds or from the same or different vendors.

specification

很显然，我们将大量借助 QingCloud AppCenter 的经验来实现在多云环境下“云化”传统应用，这其中包括规范在内。为了简单起见，我们在这里展示了一个不完整的 ZooKeeper 云化规范版本，如下所示，完整版本请参考 AppCeneter 示例，该规范很容易理解，文件 cluster.json.tmpl定义了 ZooKeeper 集群信息，如名称、描述、多少节点（每个节点的 CPU、内存、存储、镜像、生命周期等），双花括号中的变量来自最终用户从 UI 输入的信息，这些变量是由文件config.json定义的。ZooKeeper 应用开发者将这两个文件打包上传到 OpenPitrix 平台，最终用户在部署应用的时候输入必要的信息、点击部署按钮即可。这个应用程序包(还包含语言文件)如同镜像是虚拟机的模板一样，它是一个应用程序的模板，只是它相比镜像更为的复杂罢了，因为它可能包括多个镜像，而且要定义应用程序集群的整个生命周期，还有，它还要支持自定义的监控、健康监测等等更多的功能。正如我们所看到的，开发者将传统应用移植到云平台中除了几个说明文件毋须任何的编码。

cluster.json.tmpl for ZooKeeper

config.json for ZooKeeper

架构

整个系统中最具挑战性的工作就是如何将上述的一套标准的传统应用软件包无需任何修改就能部署到多个云平台中，因为传统的应用均是基于虚拟机而非容器的。OpenPitrix 设计团队经过了激烈的讨论，最后总结出如下几个原则：

• OpenPitrix 自身是可以部署到任何地方的。因此，该平台必须支持最终用户同时将应用程序部署到公共云和私有云中，这意味着 OpenPitrix 从架构必须支持能够通过公共互联网部署应用程序；
• 针对每个 IaaS 平台供应商的架构是一致的；
• 最终用户部署应用程序毋须任何麻烦的配置，这意味着对于最终用户而言，一切都是透明的，例如元数据服务的初始化。

经过反复的讨论后，我们提出了以下解决方案。首先解释一下架构中出现的一些术语。

• Drone：该组件由 agent 和 confd 所构成，Confd 是自动配置 app 实例的守护进程，agent 负责和 Frontgate 的通信，例如来自最终用户发来的指令。agent 也会负责根据应用的状态启动和停止 confd。Drone 是预先安装到应用程序所在的每一个镜像中。
• Frontgate：该组件包含 proxy 和 etcd，Etcd 是后端的存储，存储集群的信息包括集群名称、描述、多少节点、由最终用户发来的指令等。Proxy 连接 Pilot 和 Drone 的 agent，因为虚拟机是运行在 VPC 中，Pilot 无法和 Drone 直接进行通信，所以需要在它们二者之间设置一个 proxy。在有应用运行的 VPC 中会存在一个 Frontgate，这个 VPC 内的所有应用集群共享这个 Frontgate，这个 Frontgate 是当最终用户第一次部署应用程序时，系统自动创建的，且 Frontgate 对最终用户不可见。
• Pilot：接受来自集群服务的指令和信息的组件，如创建集群等，并可以传递指令给 Frontgate，它还接收来自 Frontgate 上传上来的信息。

毫无疑问，工作流程很复杂。我们以部署一个应用程序为例来进行简要的说明，一般步骤如下：

1. 最终用户发送请求到 API 网关，API 网关会将之转发给相应的服务，如集群服务；
2. 集群服务收到请求之后，将之放到 job 队列，job 控制器收到后将其分解成 task，然后放到 task 队列；
3. Task 控制器将 Task 从队列中提出，然后去调用特定的云平台 API，创建集群资源如主机；
4. 集群服务同样会给 Pilot 发送集群的元数据；
5. Pilot 转发信息给 Frontgate 的 proxy，然后 proxy 会将信息注册到后端的 Etcd；
6. 应用程序镜像内部的 Drone 的 Confd 会监测 etcd 的更新，然后自动更新应用配置；
7. 最后，Drone 中的 agent 为用户启动应用。

整个流程，如下图所示：

应用程序镜像

正如上面所阐述的，传统应用通常是部署到虚拟机中的，而微服务应用往往是进行容器化的。如果我们使用虚拟机来支持传统应用部署到多云环境中的话，会遇到非常大的挑战。这意味着应用的开发者或供应商在每一个环境中都需要上传或创建镜像，而这在现实中往往很难行得通，尤其是公有云供应商提供很多的 region 和 zone。鉴于此种情况，我们选择了在虚拟机中运行容器的方式来解决此问题，可以建立集中的容器镜像仓库，需要运行时直接拉到云平台中的虚拟机中即可。此种方式也有弊端，那就是在网络不好的地方，拉容器镜像就是一个大的问题，如果下载容器镜像的时间过长，对于用户的体验和耐心都是极大的挑战。要解决此问题，在每个云环境部署缓存或 mirror 是非常好的选择。第二个问题是开发人员需要知道 Docker，这似乎不是一个大问题，因为在我看来，掌握 Docker 是当今开发人员的基本技能。

但是，我们依然需要支持直接在 VM 中安装应用程序软件，并将 VM 分发到每个云环境中，原因如下。

• 有一些应用需要定制化 Linux 内核；
• 开发者没有掌握 Docker 技术，或者说因为额外的学习或开发成本，开发者并不愿意用 Docker 的方式将传统应用移植到云环境中；
• 最终用户没有意愿去创建一个诸如 Kubernetes 这样的容器编排集群去部署一个传统应用，或者是用户没有 Kubernetes 环境。

在第一个版本中，会在虚拟机中使用 Docker 的方式来解决应用程序镜像的分发问题，在下个版本中，将支持直接在 VM 中安装应用程序软件。

如何支持多种类型的应用

正如我们开始所说的，OpenPitrix 是支持多种类型的应用程序。当一个用户开始部署一个应用的时候，系统会自动识别应用类型并选择相应的插件去创建应用集群。我们用 provider interface 和 provider plugin 来支持不同类型应用以及将来可能出现的未知应用类型。系统目前有两大类应用类型：基于虚机的和基于容器的，前者是支持传统应用上云，这个我们已经在上一章节解释过了，现在我们解释一下 OpenPitrix 如何支持日益流行的基于互联网规模的应用程序：微服务和 Serverless。对于 OpenPitrix 的第一个版本，没有打算支持 Serverless，我们认为 Serverless 应用还没有标准，而且也没有看清楚谁将赢得最后的胜利。市场上没有与其他主要供应商兼容的产品或服务。那么这样的局面就是家家都是标准的制定者，所以，OpenPitrix 团队决定第一个版本暂时先不支持将 Serverless 应用部署到多云环境, 并持续关注 Serverless 最新标准的进展。

另外，在容器的编排方面，开源项目 Kubernetes 已经赢得了胜利，因此，OpenPitrix 会支持 Kubernetes。Helm 是一个非常不错的 Kubernetes 包管理器，在第一版 OpenPitrix 中，我们使用 Helm 作为 Kubernetes 应用程序的部署工具，在下一版或许会直接采用 Kubernetes 的应用编排格式。

As shown, we will Helm client packaged as shown below to OpenPitrix, additional server-side component Helm Tiller will OpenPitrix as a component, such as an end user OpenPitrix, and do not need to learn anything about Helm, we can deploy applications to Kubernetes in, however, at the time of writing, Helm services for multi-cluster management Kubernetes still room for improvement, as detailed in aND DELINQUENCY  1586 and 1918 , therefore, before the issue is resolved, we will ask each user to install Tiller a Kubernetes cluster. When an end user Kubernetes cluster configuration, the system will automatically configure and test the connectivity between OpenPitrix and Kubernetes cluster. Then, the user can Kubernetes deploy your application to any cluster of.

to sum up

Our plan for the future of OpenPitrix will add more features, such as:

• Workbench for developers, allowing developers here in a minimalist way to develop applications;
• Developers can view the status of their application, including all statistical information;
• Metering and billing functions
• Various reporting functions
• ......

OpenPitrix will be gradually developed into an application under cloudy environment management system with comprehensive solutions.