Write in front
To start a peripheral tour (hereinafter referred to as departure) is a well-known online travel site in China that focuses on "peripheral tours." Spring Cloud completed the transformation of the company's internal business microservices, and in 2019, it realized the migration of Spring Cloud to the UK8S platform.
This article introduces our UK8S-based Spring Cloud transformation practices in terms of business architecture, Prometheus JVM monitoring, peak elastic scaling based on HPA, APM link tracking based on Elastic, and Istio service governance.
Why K8S & Why UK8S
As the current mainstream microservices framework, Spring Cloud defines a series of standards for service governance such as intelligent routing, fuse mechanism, service registration and discovery at the functional level, and provides corresponding libraries and components to implement these standard features. Serving the surrounding environment provides the greatest support.
Before the transformation, Spring Cloud's business architecture is as follows: the service discovery part uses Spring Cloud's Eureka component, the fuse component uses Hystrix, the service gateway uses Zuul and Spring Cloud Gateway (historical reasons), and the distributed configuration mainly uses Spring Cloud Config (part of the team used Apollo), and achieved load balancing on the customer service side through Feign.
But Spring Cloud also has some inevitable shortcomings, such as the high application threshold and learning cost brought by different components based on different frameworks, and the need to control many components at the code level, which runs counter to the goal of microservice multilingual collaboration.
Internally, due to historical reasons, the API gateway architecture used by different groups is not uniform, and there are multiple sets of Spring Cloud, which causes inconvenience to unified management; Spring Cloud cannot achieve gray-scale release, and it also brings certain to the company's business release. inconvenient. More importantly, as a peripheral travel website, we often hold some promotional activities, facing the demand for elastic expansion and shrinkage of resources during peak business periods, and relying on Spring Cloud alone cannot achieve resource scheduling to meet the needs of automatic business expansion and reduction. .
When deciding to transition to UK8S, we also considered using Kubespray to build a K8S cluster and connect the K8S cluster to cloud resources through Cloud Provider, such as using Load Balance, Storage Class, Cluster Autoscaler (CA), etc. In this case, the newly added Node requires separate deployment and installation of Cloud Provider, which brings certain complexity to the operation and maintenance work.
UK8S realizes seamless connection with internal UHost cloud host, ULB load balancing, UDisk cloud disk and other products. We can easily create and call the above products within the UK8S cluster. In the scenario of peak resilience, the CA plug-in inside the UK8S can also be used to automatically expand and shrink Node-level resources, which greatly improves the efficiency of operation and maintenance. Through its CNI plug-in, UK8S is connected to UCloud's own VPC network, eliminating the need for other open source network solutions, reducing network complexity; and the unique nature of UK8S's non-encapsulation, which also gives more room for transformation and can fail When you quickly check and solve the problem yourself.
Overall business architecture
The process from Spring Cloud to UK8S is also a process of reorganization and unification of internal service modules. In this process, we made the following changes to the overall business architecture:
1. Remove the original Eureka and use Discovery under the Spring Cloud Kubernetes project. Spring Cloud's officially launched project Spring Cloud Kubernetes provides a common interface to call Kubernetes services, allowing Spring Cloud and Spring Boot programs to run better in the Kubernetes environment. In the Kubernetes environment, ETCD already has the necessary information for service discovery. There is no need to use Eureka. Through Discovery, you can obtain the list of services registered in Kubernetes ETCD for service discovery.
2. Remove Feign load balancing and use Spring Cloud Kubernetes Ribbon instead. There are two types of Ribbon load balancing modes: Service / Pod. In Service mode, you can use Kubernetes native load balancing and implement service management through Istio.
3. Gateway marginalization. The gateway is used as the original entrance. All the removal requires a large-scale transformation of the original code. We deployed the original gateway as a microservice in Kubernetes and used Istio to manage the traffic entrance. At the same time, we also removed fuses and intelligent routing, and implemented service management based on Istio as a whole.
4. The distributed configuration Config is unified into Apollo. Apollo can centrally manage the configuration of applications in different environments and different clusters. After modification, it is pushed to the application side in real time, and has the characteristics of standardized permissions and process governance.
5. Increase Prometheus monitoring. In particular, it monitors some parameters and some defined indicators of the JVM, and implements HPA elastic scaling based on the monitored indicators.
After Kubernetes, the business architecture separates the control plane from the data plane. Kubernetes Master is naturally used as the control plane to realize the control of the entire set of business without deploying any actual business. The data plane contains projects based on different languages or architectures such as Java, PHP, Swoole and .NET Core. Because different languages have different requirements for machine performance, we deploy various projects on Node nodes with different configurations through the Node Label in Kubernetes, so that applications do not interfere with each other.
JVM monitoring based on Prometheus
After the Spring Cloud is migrated to Kubernetes, we still need to obtain a series of low-level parameters of the JVM to monitor the running status of the service in real time. Prometheus is a relatively mature monitoring plug-in, and Prometheus also provides a Spring Cloud plug-in, which can obtain the underlying parameters of the JVM for real-time monitoring.
We set a series of detailed parameters such as response time, number of requests, JVM Memory, JVM Misc, Garbage Collection, etc., to provide a reliable basis for problem solving and business optimization.
Peak Elastic Scaling Based on HPA
To start as a peripheral tour service ordering platform, in the business process, it often involves sceneries that require peak elasticity such as scenic spots and hotel ticket buying. The HPA function of Kubernetes provides a good way to achieve elastic scaling scenarios.
In Kubernetes, HPA is usually implemented by Pod's CPU, memory utilization, etc., but in Java, memory control is implemented by the JVM. When the memory usage is too high, the JVM will reclaim the memory, but the JVM will not return to the host or Containers, it is not reasonable to expand and shrink the cluster based solely on Pod / CPU indicators. We use Prometheus to obtain the http_server_requests_seconds_count (requests) parameter in Java, convert it to a parameter recognized by the Kubernetes API Server through the adapter, and dynamically adjust the number of Pods in real time based on this indicator.
The UK8S product also provides its own cluster scaling plug-in. By setting the scaling group and matching the corresponding scaling conditions, it can create the corresponding cloud host as a Node node in time, which is convenient for us to pull up resources more quickly and efficiently during peak business hours.
Elastic-based APM link tracking
Under the microservices framework, a request often involves multiple services, so service performance monitoring and troubleshooting becomes complicated; different services may be developed by different teams or even implemented in different programming languages; services may be deployed in several Thousands of servers across multiple different data centers.
Therefore, you need tools that can help you understand system behavior and analyze performance problems so that you can quickly locate and resolve problems when a failure occurs.
There are many open source APM components on the market, Zipkin, Pinpoint, Skywalking, etc. We finally chose apm-server based on Elastic open source. It is because there are too many open source monitoring projects on the market, but the projects cannot communicate well. Elastic collects business logs through filebeat, monitors application service performance through metricbeat, implements tracing between services through apm-server, and stores data in es in a unified manner. The barriers between them can help O & M and development and locate faults more quickly to ensure the stability of the system.
Istio service governance
Based on application security, observability, continuous deployment, elastic scaling and performance, integration of open source tools, open source control plane support, and program maturity, we finally chose Istio as the service governance program, which mainly involves the following Several parts:
1. Istio-gateway gateway: Ingress Gateway is logically equivalent to a load balancer at the edge of the grid. It is used to receive and process outbound and inbound network connections at the edge of the grid, including open ports and TLS configuration. , To achieve the management of north-south traffic within the cluster.
2. Mesh Gateway: The virtual Gateway inside Istio represents all Sidecars inside the grid, and realizes the communication between all the services inside the grid, that is, the management of east-west traffic.
3. Traffic management: After removing the original components of Spring Cloud, such as fuse and intelligent routing, we realized the function of http traffic management through a series of configuration and management inside the Kubernetes cluster. Including using Pod tags to group specific service processes (such as V1 / V2 version applications) and implementing traffic scheduling, individually defining service load balancing strategies through Destination Rule in Istio, and redirecting based on source services and URLs to achieve target routing offload, etc. , Through MenQuota, RedisQuota to limit the flow and so on.
4. Telemetry: Obtain telemetry data through Prometheus to realize the monitoring of gray project success rate, east-west traffic differentiation, service peak traffic, and service dynamic topology.
to sum up
At present, we have migrated all of our "cloud guest praise" social e-commerce apps to UK8S, and the development languages include Java, PHP-FPM, NodeJS and so on. Combined with CI / CD, it can quickly realize service iteration and new projects go online, greatly improving the efficiency of development and operation and maintenance; through a perfect log, monitoring, link tracking and alarm system, you can quickly locate faults and based on telemetry data Pre-judge the peak in advance, achieve automatic service scaling through HPA, and scientifically allocate resources, which greatly reduces the cost of computing resources; through Istio service governance, traffic management is well achieved, and grayscale release is easily achieved based on this.
Next, we will further enrich the CI / CD pipeline, adding unit testing, code scanning, performance testing, etc. to improve test efficiency; introducing chatops to enrich operation and maintenance means; using Istio to achieve multi-cloud management to further ensure business stability.
◆ The author of this article: Wang Qiong, "Operation and Maintenance Architect and Operation and Maintenance Manager" of "To Be Around", is responsible for the company's cloud native implementation and enterprise containerization transformation. In 2016, he began to contact K8S and continued to cultivate in the field of K8S and Service Mesh. He is committed to building a production-level container service platform.