In recent years, the popularity of container-related technology, container, docker, k8s, mesos, service mesh, serverless and other terms I believe we have heard how much domestic Internet companies are all in contact and the use of related technologies.
Healthy Way as early as 2016 started the assessment of the container. Grossly now the stability and viability of container-related technology has also been a lot of validation in this context that we launched the channel container of Practice.
Of course, in practice the process, we also lack encountered some problems, we hope to record and share some of the things we encountered through words, but also want to bring a container of the road to help everyone.
Overview
Intended Audience
This series of articles will not involve a lot of k8s technique that involves only the finalization and use our program. And the article will not start from scratch, we will assume that there is a certain understanding of some basic k8s, the best target audience is being applied early k8s and ready to be applied k8s company.
Because ecological k8s very good, relevant articles and more particularly, entry and presentation articles, documents, very much, you can go on their own to supplement the relevant knowledge.
But there is a very important issue is not a complete series of production available, many for achieving analysis capabilities and technical articles k8s.
Even if there are manufacturers of Applications talking about the article but also relatively coarse grain size can only provide a small question about solution ideas also need to explore their own.
How are new k8s the study?
If we just look at the main entry k8s recommend official document (English) Do not prohibitive English by translation software can generally understand basic. The reason it is very simple: k8s develop very quickly, only official document is the most reliable, a lot of the content is already out of date.
Auxiliary, then we can go search for some of the series of articles (like scattered articles because k8s version inconsistent likely to cause trouble), I was by k8s series of courses on XX time entry, I recommend just getting started, students can start this series began . Enhance learning on the go again k8s official documents have a certain mastery of k8s later based on the content of courses.
Ali cloud has recently launched a series of courses: " CNCF × Alibaba native cloud technology open class " (author XX time series are among the top oh) are also being updated, the content is also very well written can be used to enhance learning a bit.
We have done what?
Fleet (self-study, based on k8s applicable to the Company to build a container operation and maintenance and continuous integration system)
Upgrading the company's micro Services Framework (compatible with the original micro-service call after the introduction of k8s)
Production line to build high availability clusters k8s
Server resources (CPU, memory) when tension frequently and avoid downtime optimization
BGP ClusterIP availability and interoperability PodIP and open through the switch within the network and the existing network cluster k8s
Global container hosts
Java8 (JVM) tuning and optimization in the container
PHP containerized
Simple monitoring and alerting (Prometheus + Grafana)
Technology Selection
- Filing system: kubernetes (current version 1.14.1)
- CRI: Docker (current version 18.09.4)
- CNI: Calico (current version 3.7.2)
- Image Registry: Harbor (current version 1.7.5, 1.8.0 ready upgrades)
- Load balancing: LVS, HAProxy
- Clusters guide: kubeadm (current version 1.14.1)
- Monitoring Alarms: Prometheus + Grafana
- Fleet
- Backend: Java8 + Spring family bucket + fabric8io kubernetes-client
- 前端：TypeScript + React + Ant Design Pro
Select the deployment of k8s
Before we have considered two options for deployment k8s cluster.
- Physical machine + k8s
- VM + k8s
We chose the virtual machine + k8s.
Because we feel that the virtual machine to bring the loss that we can live with. Some performance loss in exchange for good maintenance which is what we want.
Our current situation
Since the early selection of the right technology, the company carried out a micro architecture services development has gone through a very long time.
So most of our applications are stateless. This migration k8s brought a lot of help (the total is well known that there are state of the application is very difficult to migrate, also in k8s also seemed more trouble) for us.
So if most of the applications we have now or state, you can consider the application reconstructed considering a move to k8s.
Development line (overall migration in progress)
In development line we basically can migrate to k8s, is gradually recovered resources, the resources of the recovered gradually added k8s, the currently running about 80 days, the last month before failure (development line resource constraints, because resource constraints triggered a node goes down frequently pits, there will be a dedicated space to share with you in detail later), it has been stable operation for about 30 days.
Currently, our lines of development k8s cluster resource because the relationship is not highly available.
a master station (ETCD stack)
node Wutai
a harbor
Total 7 virtual machine
Application
Server resources
Production line (edge applications migration)
In our production line maneuvers part of the resources used to build highly available k8s cluster environment.
We also conservative in the action line, currently a small part of the edge of the application migration (non-core) to the production line. The company RP micro-services component uses TCP long connection also stepped on a load balancing optimization pits are also being compatible in. There will be detailed later in length to illustrate the problem.
As the production line to build a more adequate resources have not been k8s complete failure (in the middle because the network policy configuration errors led to a short service inaccessible).
It has been stable operation of 53 days.
Three master
node four units
etcd three
Two harbor
Two lvs
Two haproxy
A total of 17 virtual machines
Application
Server resources
Our lack of things to do and follow-up
We do not currently use CSI to relevant content, we currently do not support stateful applications. We will consider the establishment of follow-up Ceph cluster to add this capability.
We currently do not use Ingress ability (we are currently using ClusterIP Service), the follow-up according to the degree of urgency might consider joining Ingress ability.
We currently do not have a log collection (currently the program is dependent on their own view or log transmission logic diagnosis through the console logging function WebShell Fleet system)
Our current build system is relatively fixed, not so flexible, the follow-up may introduce Drone third-party build system.
.net core containerization
There are no series outline? What will be the next to share?
I'm sorry, I'm really not so strong global outlook to tease out all share the outline. I will share based on the approximate order of our distress. I will share what one would probably at the end of each of the write.
Next should share: to build highly available cluster k8s content, detailed description will not build steps (share the contents of our current topology map, high availability testing programs, etc.), the main target audience of this series still has some basis students will go beyond that.
At last
The first is to thank. Thanks to the leadership of CTO and manager of the utmost support and coordination. Thanks to the excellent capabilities and document k8s communities.
ps: All content sharing is not necessarily entirely my results, including our little friends have been supporting this project.
If you have questions or need to communicate via the comments feature or private letter I (recommend using the priority review, comment content is an asset to the reader).
The company (Fuzhou) there are still a number of technical positions (java, big data engineer) gap, interested students can send me a message, I can help forward 66.
About Fleet
Fleet system is our company self-development of a container operation and maintenance and continuous build system.
Pending subsequent occurrences in a systematic way.
Fleet system we are planning stage.
Is currently in two ongoing (one is basically meet the company's developers to migrate to the daily operation and maintenance and use of k8s).
The following is a partial Screenshot
