Highly reliable design, Etcd cluster, Kubernetes three main nodes to ensure high availability cluster.
Based on GlusterFS / nfs cluster, providing storage volumes service in a production environment and non-production environments.
Flannel + VXLAN, provide reliable network services.
It provides distributed monitoring and unified logging center, a full range of application of regulatory capacity.
Support Redis / ZooKeeper and other common middleware cluster layout program.
Universal Component Services, provides the ability to quickly build enterprise-class applications.
In this architecture, we can be divided into four layers, as follows:
User level: i.e. normal user.
Service Access Layer: haproxy based gorouter and service discovery, load balancing and routing distribution.
Application middleware layer: Common-based middleware redis / zk such as the use to the business.
Business application layer: the building and running enterprise business applications, such as CI / CD continuous integration, micro services, alarm monitoring and log management, warehousing and other services based on proprietary image Kubernetes platform.
I.e. laas base platform layer supports consisting of a container Kubernetes GlusterFS data management platform and persistent storage systems: infrastructure layer.
Infrastructure Layer
Kubernetes platform
Deployment Manager: Kubernetes platforms are over Ali and so on, and their own private clouds building maintenance.
Communication network: between the container and the container, the container and the host network, can be used or the like Calico Flannel embodiment, we use Flannel.
HA HA: Kubernetes Master and Node node is divided into two types of nodes, the former responsible for the operation of control-related cluster management service, which is responsible for running Pod container. Node node in multi-mode, due Kubernetes Pod has a natural disaster HA redundancy to achieve high availability, therefore, we do not need to care about HA HA Node node, but only to care HA, Master node Master node and HA, nginx scheme via several Master node +. From Kubernetes 1.12 version, kube-proxy service default ipvs achieve the abolition of the previous iptables. This helps improve performance and stability under K8s large-scale cluster environment.
Docker and optimize the operating system: In a production environment, Docker and operating system versions should use the newer release version. Also, the host operating system should do some degree of optimization.
GlusterFS / NFS datastore
Kubernetes persistent data storage platform can be used GlusterFS, NFS and other storage solutions. Wherein, the GlusterFS applicable to the technical background or a large-capacity storage requirements of the company; and NFS adapted to store a relatively small capacity requirements, non-professional storage company technical background.
Business application layer
Image Management: Using private mirroring Harbor warehouse services.
Log Management: Using Elasticsearch, Filebeat + self-development interface.
Monitoring alarm management: Use open-falcon + self-study, also used the Prometheus + self-development technology stack.
Micro-service architecture: Use Istio service program Service Mesh grid, mainly gray dissemination.
Devops: self-development pipeline pass integration platform;
Application Type: Stateless class service use deployment, using stateful class service Statefulset, pass using the helm.
Planned Namespace: According to the project - environment specified namespace, unified management, such as monitoring pod, service, pvc and other resource alarm and log management. Thus, various applications can more easily manage and differentiate on K8s.
Application middleware layer
For stateful applications, the online environment only on the redis, zk, mq common middleware and so on, because the database requires a strong stability and reliability, being unused.
After the container middleware, additions and deletions can be carried out on the pass, easy management and maintenance services, but also to avoid the application delivery cycle longer.
Service Access Layer
Easy to maintain vessel expansion middleware management.
The entire data flow is shown below, load balancing and routing rules, domain distribution and use gorouter haproxy achieve, gorouter simply is a router, haproxy is load balancing. In addition, gorouter and haproxy achieve HA high availability, it is conceivable in K8s cluster, a large number of import and export traffic haproxy been to gorouter and its load is very large, its importance is self-evident, and therefore it is very important to achieve HA. We usually deploy at least more than two nodes.