[Interpretation of the latest technology at DockerCon2017] How to deploy a highly available Kubernetes cluster in Alibaba Cloud with one click

Abstract: In Yunqi TechDay 34: The latest technical interpretation of DockerCon2017, Xie Yaoyao, a technical expert of Alibaba Cloud's container service team, shared the title of "Kubernetes on Alibaba Cloud", mainly introducing the basic concepts of Kubernetes and how to quickly One-click deployment of highly available and secure Kubernetes clusters and the capabilities provided by Alibaba Cloud's Cloud Provider.

Abstract: In Yunqi TechDay 34: The latest technical interpretation of DockerCon2017, Xie Yaoyao, a technical expert of Alibaba Cloud's container service team, shared the title of "Kubernetes on Alibaba Cloud", mainly introducing the basic concepts of Kubernetes and how to quickly One-click deployment of highly available and secure Kubernetes clusters and the capabilities provided by Alibaba Cloud's Cloud Provider.

The following content is organized based on the live video and shorthand of the speakers.

Today, I will mainly share three issues:
Introduction to the basic concepts of Kubernetes
How to quickly deploy a highly available and secure Kubernetes cluster
on Alibaba Cloud Kubernetes Cloud Provider in Alibaba, that is, some capabilities provided by the integration of cloud vendors

First, a brief introduction to Kubernetes. Kubernetes is abbreviated as K8S, which is a community name. Kubernetes is a part of the entire CaaS workflow. What is CaaS? It is Container As A Service - container as a service. The right picture in the figure below is a simplified version of the architecture of CaaS. The bottom layer is a basic service, and the upper layer mainly provides some comparisons such as physical machines, networks, and storage. hardware infrastructure. A PaaS platform will be built at a higher level. The main objects delivered in the PaaS platform are some virtual machines, or logical structures, such as logical VPC networks, storage, etc., so that you can easily create these resources with one click without You need to go to the computer room to operate and manage these things yourself. Then, a layer of container platform will be built on the PaaS platform, and the main interaction unit in this layer is the container.
176b7cbce6ea004780b019541e18665a9c0073e0
For container services, at this layer, you don't need to care about the features of the underlying PaaS platform. For example, when creating an application, you don't need to care what the underlying virtual machine is, or whether it is You need to manually create a virtual machine network and storage, and then build and place the application to run. At this time, you only need to declare the application first, and then tell it that I need to run my application in this CaaS platform, and then you don't need to do other things Worried. The platform will actively help you find the most suitable machine and the most suitable environment, deploy applications, and help you solve application failure recovery, network access, and resource allocation problems. Above this level of Kubernetes is the green part in the figure above. It belongs to the orchestration system, which can easily distribute applications to each node according to the appropriate method of the system, and then use it to handle resource scheduling issues such as failure recovery and copy guarantee. In addition, it also provides life cycle management of containers, such as starting and stopping an application, restarting when the application fails, or performing operations such as deletion after the application is not needed.

Next, I will share some basic concepts of Kubernetes. A core concept is Pod. Pod is a collective name for a group of containers. It uses a group of containers as the most basic scheduling unit. This group of containers will be coordinated together. For example, if there are two containers belonging to the same Pod, they will be scheduled to the same node, scheduled together, and Fail Over together. Service is an abstraction of a group of Pods. For example, when creating an application, you may need to create many Pods to provide services to the outside world at the same time. Then this group of Pods is at the same level and has the same status. A Services pair is provided in front of it. This group of Pods is used for load balancing. When accessing the Services, a Pod will be randomly selected from the backend to provide services to the frontend, which is equivalent to the concept of load balancing, but Services are internal to the cluster, and its network will only confined to the cluster.

The other is the concept of Deployment, which is a descriptive concept, that is, when describing an application, it is necessary to declare the state that the application needs to achieve. For example, it needs to have three copies running all the time, and it needs to be automatically pulled up in the event of a failure. . Use Deployment to describe it, and then the system will automatically guarantee these things. Once something does not meet expectations, the system will automatically convert this state to normal, whether by recreating a set of Pods or pulling up the original Pod. Another important concept in K8S is persistent volume. Generally speaking, the container does not have a data volume when it is created. For the data stored in the container, if the container is deleted, the data will also be will be lost. K8S provides a concept of persistent volume, that is, when creating a service, you can create the volume first, and then mount the volume into the container. In this way, when the container is deleted, the persistent volume will not be deleted, and the data can be retained.

The following figure is a schematic diagram of the architecture of K8S, which is a typical stateless, multi-Master distributed architecture diagram. The main functional components of K8S: Kubelet, equivalent to Agent, provides a daemon service on each node, reports node status, runs Controller, etc. aplserver is a Master, it is stateless, and all data is stored in the back-end consistent storage. The Scheduler is responsible for scheduling all Pods, that is, applications, finding a suitable location for the application and running it.
aa5868d1fbe7d22c170fa5ca87fc7248bccc8634
Here, a Load Balancer is used at the front end to provide a highly available load balancing service. After one of the masters hangs, there will still be several other masters providing services, so don't worry about several masters hanging up.

One of the biggest problems encountered in the use of K8S is the deployment problem. As we all know, K8S has been criticized in the industry for being too difficult to use, with too many components and too many concepts, making it incomprehensible. For developers, they don't want to worry about how to quickly deploy the cluster or how to understand the underlying things. They only need to be able to deploy a cluster when it is used. This is the ultimate goal that needs to be achieved, that is, the deployment is simple.
9c115afe0372e5c1c59223a5c08dec066dd03b9a
Take a look at some of the more popular deployment tools now. Kubeup is the earliest deployment tool. It supports a variety of environments, such as CentOS, Ubuntu, J2EE, AWS, etc., but there is currently no way to deploy it directly on Alibaba Cloud. Some manual changes. The bottom layer of Kargo mainly uses Ansible, which is suitable for people who are familiar with Ansible to deploy and use. Kops is highly integrated with cloud vendors such as AWS, so it is easier to use under AWS. Kubeadm is mainly natively supported by K8S, and it is also a way to be officially promoted in the future. But this does not prevent us from using it to deploy a HA K8S cluster.

So what is Alibaba Cloud's choice? ROS + Kubeadm for one reason: simplicity. First of all, let’s briefly introduce ROS. ROS is Alibaba Cloud’s resource orchestration service. It can easily create various resources required by Alibaba Cloud with one click, such as ECS, VPC network, SLB and other resources, etc. Please refer to relevant documents. You can go to Alibaba Cloud official website to check.

One-click deployment is mainly divided into three steps:
enter the ROS console, there is a Kubernetes for chinese version, and then click to create.
Select a created area.
Fill in the created cluster name, fill in the node password, and click Create.
This is one-click creation, very simple.

The main deployment process is divided into three steps: the first is to create a cluster, you need to provide the KeyID and KeySecret of Alibaba Cloud, first you need to set these two environment variables, and then you need to download a script from the official website of Alibaba Cloud to execute a node- The type=Master command, this command needs to be executed on the Master. After the execution is completed, a Master is initialized on this machine, and then it will return a Token and the IP of the node. The second step is to go to the node node and change the node-type to node, and then bring the KeyID and KeySecre, as well as the End Point and Token just returned, and you can add this node to the Master cluster you just created, and so on. You can add more nodes to the cluster. When you don't want a node, you can also destroy the node with the node-type = down command.
What fc833a1947a857c67c29bfa235326245e81bdf7b should
pay attention to is that the consistency of hostname must be ensured in the process of creating a cluster. The CP specification uses hostname as a parameter for checking the consistency of ECS, so make sure that the hostname is the same as the value passed to the CP. Another problem is Alibaba Cloud's security group settings. The security group prohibits access to all non-VPC addresses by default. Therefore, the security group needs to be opened according to the minimum requirements. The CP uses the 172.16 and 172.19 network segments, and these two network segments need to be set Open it, and you also need to open the host ports 30000 to 32768 to provide services to the Service. The third more important issue is the default kernel configuration of CentOS, which requires the configuration of the kernel's net.bridge.bridge-nf-call-iptables=1 parameter, otherwise the data traffic of the bridge on all hosts will not pass through the iptables rules. conversion, resulting in no way for services to access each other. If there is no way to solve the problem, you can resort to the ultimate killer - tcpdump, and then capture the packet and analyze the data.
dd87e5b24c00016e6bc326fb0c6e3dc4867ae1fe
Next, let's introduce the Alibaba Cloud Provider of K8S. Alibaba Cloud's Cloud Provider supports the following features: For example, in terms of network support, it supports VPC network, Alibaba Cloud's SLB and Ingress network; in terms of storage, it supports NAS and OSS. It supports the single-cluster deployment solution across regions. The cluster can be deployed in multiple availability zones, or even multiple regions, so that when a single availability zone fails, the cluster is still available, providing a high-availability solution. You can also use the native ELK as a log solution on K8S, or you can use the Fluentd-pilot open source solution provided by Alibaba Cloud, which can provide complete docking with Alibaba Cloud's log service, which is also a very good choice; and monitoring is also You can monitor the status of node nodes through Alibaba Cloud monitoring, or you can choose to use native Heapster + inluxdb as a network monitoring solution; in terms of application warehouses, you can use the current open source Helm/tiller method.
d76fc612e999810b3cbc25cae0ff3535b6c0d427
Then let's take a look at our VPC network support. Kubernetes supports a very rich network form, such as VXLAN, Calico, and VPC networks. VXLAN is mainly an overlay. After that, it is routed to the opposite host through the host's network, and then decapsulated. After decapsulation, it is placed in the kernel stack of the host, and these protocols are processed and then decoded, and transmitted to the network in another Container container. This The process involves the encapsulation and decapsulation of a packet, so its performance will suffer slightly.
cd6d19b535d3f195d396b26c5f369f5460419563
The Calico network is implemented through Layer 3 routing. It directly points the gateway to the host IP of the peer node, so that all packets sent to the container IP on the peer node can be directly sent to the peer node first. The peer node is then routed directly to the container. The benefit of this approach is that there is no additional encapsulation of packets, so the performance penalty is very small. However, it requires that all hosts are reachable at Layer 2. This requirement is usually unsatisfactory for many cloud vendors, but self-built computer rooms are generally available. Then there is the VPC network. The Alibaba Cloud VPC solution is very similar to Calico. It also uses routing to transfer the data packets of a container on one node to a container on another node, so the performance loss is very small. It is recommended for everyone to use it. VPC network, and K8S supported by Alibaba Cloud only supports this kind of network. Of course, you can also use the VXLAN network, but the VPC network will provide you with better performance, and it also provides a tenant isolation level network environment, the network segment can also be freely divided, and if multiple switches are created in different available Within the zone, an Availability Zone-level Fail Over can also be provided.

Many problems in the practice process come from customers wanting to know how the performance of the VPC network is. Here, I tested the VPC performance through several different schemes, mainly netprof, iprof and qprof. The following figure shows the results. From these results, the gap between the container's VPC network and the host's VPC network is very small, and there is almost no performance loss. Of course, the results measured by different people in different environments will be slightly different. have a test.
54b06ae9bb8bb600891b0c0c9383c35280430c21
Alibaba Cloud's CP also supports its own LSB. The access scope of the aforementioned Service is only within the cluster. If you want to access the services provided by the cluster from outside the cluster, you need an entry point. Alibaba Cloud Load Balancer can support this function. This function is also integrated into Alibaba Cloud's Cloud Provider. When using it, you only need to specify the type of service = Load Balancer in the yaml File of the application creation, and then you can create an Alibaba Cloud Load Balancer. Balancer, then it will automatically associate Load Balancer with the back-end application server, and then provide users with a static IP external network access port, so that they can access services within the cluster through IP, and also provides HTTPS. Support, users can upload their own certificates to provide HTTPS services, and can also customize the bandwidth of SLB, health check type, URL, charging type, etc.
5246c26d6a7e3fb2f449ed6102a5e1b7168bb5f5
K8S's Service is within the cluster, and Alibaba Cloud now provides another solution that uses Ingress. Ingress can bring in external traffic, so what is the difference between this and the LSB mentioned above? In fact, LSB is a four-layer thing, which performs load balancing at the IP layer, while Ingress supports seven-layer load balancing. It can provide a simple routing service, such as directing the url of a domain name to the backend of a Service, and also Fanout, which can provide a simple service, can use a domain name to provide services, and locate different backend services according to different URLs. Users can also use multiple domain names to provide services, which is similar to the way virtual host routing, different domain names will be routed to different backend services.
74dda4f759751b9d9473a0e96333ddbcfa6e2cfc
Then the one on the right is a simple architecture diagram. The top layer is the SLB of Alibaba Cloud. The traffic of the SLB of Alibaba Cloud will be located in two Ingress Controller Pods. These two are a simple Nginx implementation. Through this Nginx Do layer 7 load balancing, and then relocate these traffic to your application Pod. Another feature is that it supports rich Annotation, which can support SSL, third-party authorization, ORL's Rewrite whitelist management and other functions.

Alibaba Cloud's Cloud Provider also supports various storages, such as NAS storage and OSS storage. NAS storage is backed by the native K8S NFS driver. Here is a brief introduction to the use mechanism of KBS's Volume. K8S's PV is a persistent volume, which needs to be created by the cluster administrator as a resource in the cluster. If the application side wants to use it in the application , you must first create a PVC, and declare that you want to use the storage resources of the cluster, and then K8S will select the most suitable and most suitable ones from all the cluster volumes, and bind them together. Binding means that you can use This is PVC. For applications, it is necessary to declare the need to use PVC in the Pod, and then the application is bound to the PVC, so that when the application is created, K8S will mount this to the application Pod, so that it can be written like Write Persistent-Volume like local disk.
01caceef542b8cfcd890fad3ee54097d7858e437 Alibaba
Cloud also supports multiple availability zones, mainly to achieve Fail Over in availability zones. Allows users to create clusters in multiple availability zones, and services are still available even if other availability zones fail.
16fe626fbe9d99c294ef3310eefbb868b76be3ee
Alibaba Cloud also provides cross-region support. The Availability Zone level is between each Availability Zone, and the networks of each Availability Zone under the same Region are connected, so it is very simple to deploy. However, the network between each region is blocked. It is necessary to connect the network of each region through the high-speed channel of Alibaba Cloud, and then deploy the application. Then there are some restrictions. By default, Alibaba Cloud resources across regions cannot be used universally, that is to say, only the resources of this region can be used in a region, so it is usually recommended to create the same resources in each region, for example SLB, after creating an application, schedule the Pods of these applications into multiple regions, and then create an SLB in each region and bind it to the application. What is the benefit of this? Bind to SLB, and then provide a smart DNS resolution at the outermost end, and import traffic to these two regions respectively. This can support a kind of adjacent user routing. For example, there is a region in Hangzhou, and some users are also in Hangzhou. , then users in Hangzhou will be guided to the Pod node in Hangzhou, so that its user response speed is very fast, and then users in Europe can all be guided to the cluster in Europe, so this option for neighboring users is Very helpful. The following figure shows the detailed diagram of cross-Region support.
f4446ebe31521500796bb865065f90fc210a0b3b
This article is the original content of Yunqi Community, and may not be reproduced without permission. If you want to reprint, please send an email to [email protected]