This article is from the online live sharing of several senior engineers of Renyun on the IT big coffee platform on January 18.
Today we mainly discuss these aspects:
1. The positioning of the configuration center
2. Cloud-based microservices requirements for the configuration center
Three, microservice configuration principles
4. Overall Architecture of Shurenyun Distributed Configuration Center
Configuration Center for DevOps and Microservices
First of all, I want to share with you why there is a configuration center? When I was engaged in software development in the early days, there was no configuration center and no microservices.
In the past, every time the system was released, whether it was a major change or a small change, it had to go through the release process. This change is sometimes so small that it only modifies a field of a configuration file, and it must be repackaged and redeployed.
Moreover, the development and operation and maintenance personnel in the enterprise are often not the same group. When deploying, developers need to prepare deployment documents for operators. In this process, due to insufficient communication and coordination, it is often impossible to predict all problems at the first time. As a result, when the official launch, users are immediately aware that there is a problem with the launch, or the deployment is incorrect, etc., resulting in some very serious problems. as a result of.
Over time, a new term has emerged in the software engineering world - DevOps. Development and operation and maintenance personnel work together to deploy and launch products. In the era of DevOps, how to transform concepts into direct productivity through some IT means. Because the larger the changes per release, and the higher the frequency of releases, the greater the risk to the system.
Therefore, the industry urgently needs a technical means to assist development, deployment and operation and maintenance personnel to reduce this risk in the continuous iterative process. Can the configuration be a dynamic configuration? No need to do some static configuration anymore. Based on this industry risk, there is an impulse to become a configuration center. As a result, distributed configuration centers appear on the market.
Whether it is a distributed configuration center or a common configuration center, what exactly is it configuring? If you push the time forward for a while, in the era when you log in to the server through SSH to modify the configuration, the configuration center has little effect.
After 2010, microservices appeared in the industry, and the distributed configuration center officially entered the stage. why? Because of the combination of distributed configuration center + microservices, we can truly realize DevOps as we understand it.
Microservice Configuration Principles
Adam Wiggins, founder of Heroku, released a "The Twelve-Factor App Manifesto", which provides a framework for building automatic configuration using standardized processes, clear service boundaries, high portability, and scalable service configuration based on cloud computing platforms. Methodology:
1. Configuration is separable and can be extracted from microservices, and any configuration modification does not require a single line of code.
2. The configuration should be central to configure and manage different microservices through a unified central configuration platform
3. The configuration center must provide configuration services reliably and stably.
4. Configuration is traceable, any configuration history is traceable, managed and available.
In the era of cloud services, what are the requirements for configuring microservices?
First of all, it must be deployed based on image management, with its own corresponding independent configuration, and the package cannot be changed due to changes in the environment. That is, it is an immutable package independent of the environment.
Secondly, when all microservices are stored through environment variables or configuration, they can be configured at the moment of startup, or they can be dynamically modified to take effect in real time.
Microservice startup configuration
After a microservice is packaged, launched, deployed, and packaged, its configuration will be pulled from the Configuration Repository during the startup phase, registered in the registry through registration and discovery, and the configuration of the service center will be pulled at startup. to the local and become part of the application.
And in the process of service running, real-time dynamic monitoring of configuration changes, when new configuration is delivered to the microservice immediately, so that the configuration has the effect of taking effect in real time.
Location of configuration center
With the above business requirements, what kind of configuration center should we make? Shurenyun believes that this configuration center must have a consistent KV storage center. KV storage means that one K corresponds to one V, and this storage must be persistent and traceable.
It must be centralized and unified configuration, real-time push, and take effect immediately.
All configurations must implement grayscale updates and rollbacks. The so-called grayscale release means that in a microservice cluster, for example, there is an order system, and some configuration updates have been made. I want to discuss in a small area and experiment to see how this configuration affects the business. At this time, the concept of grayscale update is used.
Grayscale update means that through the Data center or static IP, a certain configuration is specified to take effect only for certain IPs, or a service in the Datacenter, and other services that are not in this range will not be affected. Observe the effect, if OK, push the overall configuration to all microservices. If the effect is not satisfactory, roll back the configuration to the original state.
Full update, grayscale update, or rollback must be a configuration that can be viewed at any time, and rolled back to a certain historical point at any arbitrary moment.
The last one is the startup of multiple clusters. The so-called startup of multiple clusters means that when we store the configuration, it must be stored in a multi-cluster environment to achieve the purpose of physical isolation.
There are also some principles, business irrelevance, Open API, and configuration validation monitoring. That is to say, the configuration center must provide APIs for third-party systems to use. The effective monitoring of the configuration is to know in real time, which services have pulled the configuration, whether it has taken effect, or what is the effect of this configuration?
The support system of the configuration center
The first operation and maintenance management system is similar to the configuration of a partial static class, and the read business is directly pulled through the configuration file at startup.
The other is the development management system, which is partial to dynamic management, which represents a program or an effect achieved by changing the configuration content in real time and taking effect in real time during the running process.
A sound configuration center should support both operation and maintenance systems.
Configuration Center's Microservice Compatibility
The configuration center must have API compatibility with some existing mainstream development frameworks. When Shurenyun is a configuration center, it is difficult to estimate what kind of development framework is used when a third party calls services? Generally speaking, it is impossible for the configuration center to be compatible with everything on the market. Shurenyun chooses important frameworks for deep compatibility.
First of all, API compatibility is made with the conventional third-party cloud service frameworks such as Spring Cloud and Ali's Dubbo. At present, at least the configuration of SpringCloud, Dubbo, Nginx, Tomcat, Logback and other aspects should be supported.
This configuration has its own characteristics, so we have selected some classic cases and common things for configuration integration, such as native support for SpringBoot, Spring Cloud, and integrated support for k8s, which is the k8s container.
Shurenyun hawk distributed unified configuration center
The distributed unified configuration center of Shurenyun, named Hawk. Hawk is built based on ETCD, which mainly solves the problem of freeing developers from complex business processes and cumbersome configuration, allowing developers to focus only on their own business code, and strip out operation and maintenance and configuration. At the same time, the risks in the process of service deployment and release are reduced.
Hawk, a distributed and unified configuration center based on the concept of the microservice system, supports various types of configurations: such as Spring Cloud, Dubbo, Kubernetes Configmap, Logback, and Linux Environment. It has a complete configuration management process, configuration implementation push, and supports multiple clusters and multiple environments. Multi-version control, but also provides rich functions such as fine-grained management of configuration, such as grayscale management, arbitrary version reset, etc.
The whole system is compatible with Spring Cloud Config of the open source community and Configmap of Kubernetes, which greatly reduces the learning threshold for users and the coupling of business to the platform. The corresponding management process also regulates the use of configuration and reduces publishing errors caused by configuration.
Features
Compared with mainstream frameworks, Shurenyun HAWK has the following important features:
-
Support configuration real-time push and real-time effective. During the running process of the program, it cannot be said that the service can be updated only when the service is stopped. It must be configured in real time and effective in real time.
-
Support multi-version management, no matter which version is configured, you can view, check and activate the historical version at any time, and publish it.
-
Support multi-cluster environment, multi-environment configuration. Through the database or back-end storage, it can achieve a system that supports SIT and UAT environments.
-
The beautiful monitoring console provides multi-dimensional Dashboard and monitoring views, and supports configuration grayscale and rollback.
-
Supports global data backup and recovery, further improving the disaster recovery capability of configuration data.
-
Provides OpenAPI, supports convenient means of multi-system integration, and supports configuration of emergency plan processing.
-
Configuration process management, complete configuration process management, to ensure that confirmation and authorization must be obtained before the configuration is issued.
-
Authentication and authorization, providing LDAP integration, and multi-role rights management.
-
Supports operation auditing to ensure that configuration operations are well documented.
-
Support multiple configuration files, Spring Cloud Config, Dubbo, Logback, Nginx, LinuxEnvironment, Tomcat.
-
Support Spring Cloud service governance configuration and control, and support Spring Cloud's own Hystrix microservice governance such as circuit breaker, Fallback, etc.
-
Seamlessly integrate Kubernetes' Configmaps and Secrets, and provide enhanced enterprise management processes.
Hawk overall architecture
The first layer of access is the gateway. The overall storage is delivered to the ETCD cluster through the Hawk Server, and the ETCD cluster is then synchronized to the platform where the K8S container runs.
A classmate said just now that Hawk is similar to Apollo. When researching and comparing, we felt that Apollo had some complicated parts due to business needs, and decided to simplify the process.
Start with simplifying the state of data migration into simple parts. For example, if a new configuration is created, or the configuration is deleted, it can be done in one step. If you do not do this, you will face several situations. Whether this configuration needs to be released tentatively in a small range, in this case, you can release it in grayscale. The status becomes grayscale, and the configuration is not allowed to be changed. Either go both ways and publish the full amount to all services. Either give up the grayscale and return to the previous state, and after giving up the grayscale, it will go to the modified state.
In another case, create a new configuration and publish it in full directly, and the status becomes the published status, which can be changed at this time. But every time you change it, it will return to the original state. Will this change be done in grayscale? Or do a release? Or have some regrets about the release and don't plan to change it? At this time, find a suitable version from the historical version, activate it, and then do a release again, covering most of the business scenarios through a few simple loops.
Configuration data state transitions
Hawk Portal is the configuration interface of the main body, and the user manages the input, addition, deletion and modification of the configuration on the interface. There will be two copies of these data, one for local storage through Mysql, and the other for direct synchronization to ETCD through Hawk Server.
Since HAWK Server is synchronized to ETCD, which means ETCD is equivalent to another database, there is no mutual copying between data, thereby reducing the risk of data loss. Persistence means that R&D and operation and maintenance do data migration in the background, or data monitoring is more secure and convenient.
Shurenyun HAWK actually has two ETCDs. One ETCD is for registration and discovery. HawkServer and Hawk Portal will be registered in it as related components. Similar to Spring Cloud Eureka, Eureka is a memory list registered in Eureka Server, and all servers in the cluster share this memory information. This process has been simplified by several people and clouds, and all information is registered in ETCD.
Because the ECTD cluster is shared, the state and consistency of the components are guaranteed. The Portal and the Server no longer register on the Server through the Portal and maintain the relationship through the heartbeat, but share the persistent ETCD to ensure that the state of the data at any time is consistent, thus ensuring the registration of the service, and Service discovery stability.
Hawk and Eureka choose different paths. Eureka is relatively heavyweight, and HAWK simplifies this configuration, simplifies the complexity of this code, focuses on improving the integrity of the system, creates a closed loop of the system, and improves the stability of services through some relatively simple methods.
Once the configuration is sneaked into the local database through the Hawk Portal, how does the registration service of the microservice realize the configuration? When Portal writes the configuration to the local database, it also synchronizes to ETCD through the service server. The information stored in ETCD is persistent data.
The configuration is pulled from etcD in real time through the Server, sometimes when it is running, and sometimes when it is started. There are two strategies for pulling at startup. The configuration is pulled at startup, and the configuration is stored locally as a static file. It is pulled at runtime, and dynamic changes take effect in real time.
In fact, there are some problems in the Web layer that need to be solved. For example, because we are not a development framework, we are going in the direction of an open source system, so we need to solve the authorization between the service and the browser.
The current practice of Shurenyun is to store some user information in the local database, but it does not use traditional session building for verification and authorization, but in the form of dynamically issuing JWT, each request is dynamically issued, according to Some information generated by my personal users, each time a request comes and goes, a new Token is exchanged, and each Token takes effect in real time and has the function of renewal, instead of the traditional Session.
Configuration center integrates third-party frameworks and class libraries
Hawk has a third-party class library integration. The operation process is as follows: the operator calls the portal of the HAWK backend through the UI, and writes the configuration to the ETCD through the Hawk Server. Other third-party services held by operators and developers can read the configuration by integrating Hawk Client.
Hawk also has a new iteration, which is integrated with Sharding-JDBC, and some trends of JDBC 2.0 are deeply integrated with the HAWK system. The service realizes the strategy of dynamically changing the sub-database and sub-table by reading the configuration center configuration in real time.
Q&A
Q1: Change the distribution table strategy in real time?
A1: Yes, Sharding JDBC 2.0 focuses on realizing this function, which is also the result of the deep cooperation between Hawk and Sharding JDBC
Q2: What about historical data?
A2: Historical data is actually stored persistently. If you want to go back to a certain historical version one day, you can always call the published history by calling. Hawk has a section called Release History. Here, each configuration has a configuration history that can be traced, and a version can be viewed or activated at any time. When activated, users need to choose whether to do a full-chain release, a grayscale release, or nothing.
Q3: Does it support ServiceMesh?
A3: At present, ServiceMesh is still a relatively new thing. If ServiceMesh is configured independently, it can actually be supported. However, ServiceMesh has a lot of features, and I dare not say that 100% can be supported. In fact, Shurenyun is also working on ServiceMesh-related projects, which means that it is still very possible in the future, and it is still necessary to integrate the ServiceMesh configuration center to make a closed loop. Currently this version supports some lighter configurations.
ServiceMesh Chinese Network WeChat Official Account (ID: servicemesh), a large number of Istio, Conduit official document translations and technical dry goods articles, welcome to pay attention.
Q4: Which steps are used in development environment and production environment?
A4: SIT and UI can actually be deployed together through storage isolation. However, it is not recommended to deploy production together, and production is recommended to be deployed independently. Production or physical isolation is the best solution.
Q5: Are configuration center and service center two systems?
A5: Currently, there are two systems, but the current thinking is temporary. The current thinking is whether to expand the configuration center a little more, not call it the configuration center, but call it, for example, a microservice service platform. In the service platform, the configuration center and the registration center are integrated as a component, and some things that have nothing to do with the business are extracted, carried on some public platforms, and integrated into the platform in the form of components.
Q6: Does open source have a local cache?
A6: At present, there is no local cache, it is directly obtained from ETCD, so it is real-time, but a cache will be added to the registry.
Q7: What information is generally configured through the configuration center?
A7: For non-business things, such as developing a system at ordinary times, or developing a third-party system, we have all the configuration files, such as database connections, governance-related things, or some foreign references, all of which can be accessed through the configuration center To configure, and this configuration is not only the configuration on the page, there is a plug-in for it, after startup, these configurations are directly imported into the configuration center, and this configuration can be delivered to the local through startup, Form a local static configuration.
Q8: What if the server is disconnected?
A8: Because the server is a multi-server cluster, if it is disconnected, it does not mean that the server is accessed through the specified IP address, but the server is accessed through registration and discovery. If the server is disconnected, it means that the If there is no heartbeat, they will try to request a new connection in ETCD, and this connection will also be registered and discovered, and other servers will be discovered, so they will connect.