content
1 Introduction
1.1 Talking about grayscale from a system transformation
At the beginning of the company's establishment, in order to quickly realize the company's strategy, it purchased a system from the market and carried out outsourcing cooperation, which is also a shortcut for most companies to start a business. With the rapid development of the company, the company has established its own R&D team, and it has become particularly urgent to reconstruct the existing system. The same problem is: leaders have urgent time requirements, research and development cannot understand the existing system, and data compatibility must be achieved. Based on such requirements, the system must be launched in a grayscale method, and loyal users must be designated to conduct online activities. Testing, selecting characteristic groups for online testing, and online testing based on traffic switching.
Driven by the above requirements, I have re-considered the grayscale. Grayscale was also involved in my previous work. At that time, I thought that this should be the operation and maintenance of a large company. Now I think about it, it is not only a matter for a large company, but also something that a fast-growing company should pay attention to. Children, especially for Internet companies.
One of the characteristics of Internet products is that they are constantly upgraded, upgraded, and upgraded. Generally, the team that adopts agile development basically maintains the release frequency of once a week. System upgrades are always accompanied by risks, the risk of compatibility between new and old versions, the risk of user loss caused by sudden changes in user habits, and the risk of system downtime.. ...In order to avoid these risks, many products have adopted the strategy of grayscale release. The main idea is to concentrate the impact to one point, and then spread it out to one side. It is easy to roll back after an unexpected situation occurs, even if the impact is controllable.
1.2 Traditional system release
Searched on Baidu: Grayscale release refers to a release method that can smoothly transition between black and white. AB test is a grayscale release method, allowing some users to continue to use A, and some users to start using B. If users have no objection to B, then gradually expand the scope and migrate all users to B. Grayscale release can ensure the stability of the overall system, and problems can be found and adjusted at the initial grayscale to ensure their influence.
In a fast-growing Internet company, Grayscale actually routes requests to our grayscale version (grayscale machine) according to set rules. For example, for APIs, there are generally the following requirements: specific users (such as test accounts), specific apps (such as test apps or cooperative apps), specific modules, and interfaces (only some interfaces require grayscale, which generally It is a modification of the API Container, and some less important APIs are used for grayscale testing), specific machines (some request IPs are forwarded to grayscale machines), etc.
In the traditional software product release process (such as Microsoft's Windows 7 release process), it generally goes through several stages such as Pre-Alpha, Alpha, Beta, Release candidate (RC), RTM, General availability or General Acceptance (GA). (Refer to Software release life cycle). It can be seen that the release stage of traditional software is from the company's internal -> external small-scale testing> external large-scale testing -> official release, and the number of users involved is also a process of gradually increasing volume.
1.3 Problems that can be solved by grayscale publishing
Our company is an Internet company, but in order to ensure the reliability of the system online, the same is true. Every time the system is launched, it goes through multiple processes of development, site, test, bate, and production. When programmers and testers are confident to push the system online, the online failure of the system may be caused by a small configuration, and the risk is unpredictable.
Every time a product is released, it is not known whether the product design can bring a better experience to the user when it is handed over to the user. Special groups cannot be designated, and the online verification and experience of the new product should be prioritized.
Based on the above problems, the problems that can be solved by Lenovo grayscale:
1. Reduce the risk of going online during the release process
2. Reduce the scope of influence, and the scope is controllable
3. Reduce the dependence on testing and reduce the data construction cost of offline self-testing
4. A specific request can point to a specific server, which facilitates centralized monitoring of logs and facilitates tracking of the complete call link
5. It is convenient to cut in the system flow
6. Easy to roll back
7. Designate specific groups to facilitate system return visits, facilitate product demand collection, improve product functions, and improve product quality
8. Ensure that the versions used by users are consistent in a stateless situation
9. Avoid service suspension to bring bad experience to users
2 thinking
There are multiple ways to solve the problems mentioned above, and different companies have different implementations. As an open source grayscale system, it is necessary to take into account the non-invasiveness of the business, and it is easy to upgrade and configure.
There are many companies that implement grayscale based on deploying a set of grayscale environments as a grayscale release scenario, which is a means to achieve grayscale at some levels, but the front-end entrance configuration is messy, not easy to maintain, and cannot guarantee online access. The configuration of the system is the same as that of the grayscale system, and the environment may also be different, and there is a risk of going online.
It is particularly important to develop a grayscale system, especially for small companies. If there is no energy to do it, it is impossible to ensure the smooth launch of the system. I also wrote a grayscale system out of this impulse and made it open source, hoping to provide a solution for small companies, and hope it can be used as a better reference for large companies.
3 System Design
This system management system is mainly developed with java technology, the server mainly uses spring+springmvc+mybatis+dubbo, the front end uses easyui; the database uses mysql; the grayscale engine is currently developed using java and lua technologies.
Project code: http://git.oschina.net/xiaoleiziemail/gray
3.1 Design Goals
1. Can solve the problems mentioned above
2. To achieve flexible configuration of the system
3. Ensure that the system does not involve business
4. To be non-invasive to the original system
3.2 Desired functions
3.3 Overall Design
The grayscale system is mainly divided into two parts: control and engine. Management users manage grayscale policies and whitelists through the management terminal, and user requests are routed intelligently through different grayscale engines.
4 Management and control platform
4.1 Grayscale management
4.1.1 Whitelist Grouping
4.1.2 Whitelist Management
4.1.3 Policy Management
4.1.4 Application Management
4.1.5 Application Services
4.1.6 Applying Grayscale
4.1.7 Service Grayscale
4.2 Log Monitoring
4.3 Rights Management
5 Grayscale Engine
At present, two access modes have been implemented. The http access is developed by lua, and the grayscale is based on nginx. The dubbo access mode is developed by java, and the grayscale load balancing is used to realize the grayscale.
5.1 Http access
It is mainly to perform grayscale for the application of the Http entry, configure the nginx load on the front end of the application, and perform grayscale selection and request forwarding according to the corresponding policy during the load process.
The entry of the http request is on nginx, and nginx will select the uri according to the configuration of the location. At this time, the request data will judge whether the current application has turned on grayscale, and then judge whether it is the grayscale of the application level or the grayscale of the service level, and then perform the grayscale according to the grayscale policy configured on the management and control platform. Currently, whitelist, small traffic, Regular expressions, business domains, weights, IP segments, regions, etc. will be improved in the future.
The management and control platform will call the engine management to execute the corresponding instructions to close, open, update the policy and whitelist data, etc. Every time nginx is reloaded and restarted, it will read the configuration application information from the gray management system call interface and add it to the cache.
In order to improve performance, data such as basic application information, grayscale policies, and whitelists are cached in the shared memory block developed by nginx. Each worker process of nginx shares this data, and the cached data is updated under the manipulation of engine management.
5.2 Dubbo access
Selected at the entry of dubbo according to the rules, dubbo-consumer implements different load balancing, and performs whitelist check and policy selection during load. The system has no mandatory dependence on the grayscale management and control platform, and problems with the management and control platform will not affect the normal operation of the system.
1. Cache Load Policy
When the system starts, the grayscale policy should be pulled according to the system configuration and stored in the memory, and the latest load policy should be obtained regularly. It is necessary to provide a policy update interface that is triggered in time.
2. Whitelist update
When the system starts, it needs to pull the whitelist according to the system configuration and save it in memory, and obtain the latest whitelist regularly. It is necessary to provide a whitelist update interface that is triggered in time.
3. Load balancing engine
Select the load balancing engine used at runtime before the system goes online, which can provide methods such as small traffic and whitelisting.
4. System Configuration
Before the system goes online, it needs to enter the management and control platform and complete the corresponding configuration. When it is started, it can be used as a unique identifier to pull the corresponding configuration.
5. Monitoring and Statistics
The system caches statistical information in memory, and uploads it to the management and control platform regularly. Monitoring problems will not affect the normal use of the system (not implemented yet).
6 Imagine the future
1. Enrich grayscale strategies, such as grayscale by region, etc.
2. Realize grayscale based on multiple access methods such as MQ and Schedule
3. Realize the collection of user data
4. Implement link monitoring requested by users
5. Provide open interfaces for operation and maintenance, improve operation methods and reduce operating costs
6. Realize a new test environment based on docker and grayscale system