Release mechanism: canary release, rolling release, blue, green and released in the end what is the difference? The key point is what? 2

ylbtech- release mechanism: canary release, rolling release, blue, green and released in the end what is the difference? The key point is what?

 

1. Back to top

1、

According to the 2017 report DevOps development, high-performance organization and inefficient organization of magnitude differences in the efficiency of software delivery. Software delivery capability Technology Organization is a comprehensive ability, involving many sectors, particularly where the release is the important part.

As a technician, you may have heard of "rolling release" and "blue-green publishing" and other terms, but many people do not know the principles behind these terms. This article attempts to summarize the current mainstream release strategy, the pros and cons of each, applicability, allows developers to architects in particular have a more clear and comprehensive understanding of modern publishing technology, so that we can according to their business context, for release strategy make the right selection and practice.

A single server publishing group

First explain the concept of single-server group, previously our machines relative shortage of resources, unlike the current cloud computing and virtualization (including container technology) so advanced, so the application of the basic machine is pre-allocated statically (assigned by the general responsible for the operation and maintenance) the original application a n lived in this machine, the next release of the application upgrade also live on a n this machine, so called single-server publishing group.

1.1 brute force release

As shown below, this publication is relatively simple and crude, a bit like our traditional software upgrade, completed mainly by hand, first under the old version V1 all out, and then send the new version to the machine up. In this way introduces outage (downtime), in a development test environment is feasible, but in a production environment release, which will directly affect the user experience, this approach is generally not recommended.

Pre-Release

After release

 Advantages and Applications

Advantage:

• Simple and low cost

insufficient:

• Service interruption affected users, a problem also slow rollback

Applications:

• Development and testing environment

• Non-critical applications (user impact facet)

• Startups are short, the dead of night to find a small amount of time to do user access

 Traffic patterns

Brute force conference to introduce the service interruption time, images from Appendix 6.1

Canary 1.2 release (single server group)

A brute-force released on the basis of a simple way to improve the release, is still the mainstream publishing a lot of growing technology organizations. Step canary released under a simplified single-server set as shown below:

Pre-Release

Starting a canary

All finished

 Practice points

1. Canary general release starting 1 station, or a small percentage, such as 2% of the server, mainly to do with the verification flow, also known as Canary (Canary) test (often called gray domestic test). Before former mining absenteeism at the mine, the first will put a canary into whether there is toxic gas exploration, canary see whether survive, canary publish hence the name. Canary simple test usually by manual testing to verify, canary complex testing needs a better system of monitoring infrastructure with, by monitoring indicators feedback, observe the health of the canary, as the basis for subsequent release or rolled back.

2. If the test passes Watkins, put the rest of the V1 and V2 versions of all upgrade version. If the canary test fails, the fallback directly canary, failed to publish.

 Advantages and Applications

Advantage:

• The user experience is small, canary publishing process problem only affects a few users

insufficient:

• Posted degree of automation is not enough, it can lead to service disruption during publishing

Applications:

• Lack of sufficient confidence in the new version features or performance

• User Experience demanding website business scene

• Lack of adequate automated publishing tools R & D

 Traffic patterns

Canary accept a small amount of traffic, then the whole amount released images from Appendix 6.1

1.3 rolling release (single server group)

Published in the Canary further optimized and improved on the basis of, is a high degree of automation of publishing, the user experience is smooth, it is the mainstream publishing organization mature technology used. Released under the rolling step simplified single server group as shown below:

Pre-Release

Release, starting a canary

Release, then sent a number of Taiwan

Until all finished

 Practice points

1. Rolling release generally starting 1 station, or a small percentage, such as 2% server, mainly to do with the verification flow, similar to the Canary (Canary) test.

2. Rolling release publishing tools require more complex and intelligent LB, supports smooth flow and replace the version of La Rula out.

3. Each release, the first version of the old V1 traffic removed from the LB, and then remove the old version, the new version of the hair V2, then the flow LB access the new version. This will try to ensure the user experience is not affected.

4. A rolling release is generally issued by a number of batches of composition, the number of each batch usually is configurable (can be defined by publishing templates). For example the first station 1 (canary), a second batch of 10%, 50% the third batch, fourth 100%. Observation interval left between each batch, or by manual verification feedback monitoring to ensure that no problem recurrence next batch, so the overall rolling release process is relatively slow (typically wherein canary time than subsequent batches longer, such as canary 10 minutes, a subsequent interval of 2 minutes).

5. Rollback is issued by the reverse process, the new version will be removed from the traffic LB, remove the new version, the old version of the hair, and then access the old version LB traffic. And publishing process as a fallback procedure generally slower.

6. Rolling release foreign term is often called the Rolling Update Deployment.

 Advantages and Applications

Advantage:

• The user experience is small, relatively smooth experience

insufficient:

• Release and slow back-off time

• Publishing tools is more complex, LB require a smooth flow and pull removal capability

Applications:

• Not interrupt the user experience of the website business scene

• There are some complex publishing tools research and development capabilities;

 Traffic patterns

Rolling release, flow smooth transition, images from Appendix 6.1

Second, the two-server group released

With the maturity of cloud computing and virtualization technology, especially in lightweight containers introduced by virtualization technology, computing resources are limited and slow application problems have been solved step by step can be done on demand elasticity. Once released assign two sets of servers, a group run existing V1 old version, set to be run on the new version V2 line, then switch the flow is accomplished by LB released, which is called the two-server group publishing.

Blue-green 2.1 release (two-server group)

Blue, green and released only for dual server group released can be considered is a simple brute-force optimized release publishing. Simplify the process shown below:

 Practice points

1. Blue Group version called V1, V2 version called green groups, one-time traffic is switched from the released through the blue group directly to LB green group, canaries and without rolling release, whereby the release name cyan;

2. Problems rollback also very direct, cut back to the Blue Group will flow directly through LB.

3. After the release of initial success, the blue set of machines generally do not directly recovered, but remained to be seen a period, depending on the specific circumstances of the observation period of time may be longer or shorter, released after the observation period to confirm no problem, you can recycle blue set of machines.

 Advantages and Applications

Advantage:

• Upgrade fallback switching and very fast

insufficient:

• Switching the whole amount, if V2 version has a problem, then have a direct impact on the user experience;

• Need twice as much machine resources;

Applications:

• There is a certain tolerance of the user experience scene

• There may be surplus or machine resources on demand (AWS cloud, or cloud self-built container)

• It will not have the complex rolling release tool research and development capabilities;

 Traffic patterns

Blue, green and publish a complete switching process, images from Appendix 7.1

Canary 2.2 release (two-server group)

A simple optimization of deployment cyan, green start publishing group pull station 1 canary, canary be verified by the total amount of recurrence. Contrast blue-green publishing, publishing the advantage is to have a production flow of canary verification process, can reduce the risk of V2 may be a problem and the impact surface. Simplifies the publishing process as shown below:

2.3 rolling release (two-server group)

Rolling release is to further optimize the above blue-green and canary released in batches incremental rolling release, provide a smoother user experience.

Practice points

1. To apply for release before a new batch server, the same number and general version V1, V2 will release a new version of the application to the server. For example, if in the AWS cloud, you can apply a number of new VM API calls directly, if the container cloud Kubernetes, you can start a new batch of containers directly (using V2 container mirrored version).

2. LB will normally pull through V2 version of the machine table 1, which also corresponds to the machine canary, for flow verification.

3. Gradually released in batches to complete, each batch just need to pull through LB V2 version, and then pull out the corresponding number of V1 version. Leaving observation interval between batches, by hand or monitoring feedback to ensure that there is no problem to continue publishing.

4. There is something wrong rollback quickly, directly through the traffic LB V1 to cut back.

5. After completing the release of version V1 generally observed just in case you want to keep, such as leaving one day, no problem one day after V1 machine resources are recovered.

 Advantages and Applications

Advantage:

• User experience little impact;

• Upgrade switching and rollback (rollback) rolling release speed is faster than a single server group, LB can cut traffic;

insufficient:

• Need twice as much machine resources;

• Publishing tools is more complex, LB need to switch traffic capacity

Applications:

• Not interrupt the user experience of the website business scene

• There may be surplus or machine resources on demand (AWS cloud, or cloud self-built container)

• There are certain publishing tools research and development capabilities;

 Traffic patterns

Rolling release, flow smooth transition, images from Appendix 6.1

3. Other publishing

Above all the traditional manner of publication bias, you can cover most applications published scenarios. On-line for some of the key new features of the release, or some particular scene, there are some special publishing.

3.1 release function switch

Using the function switch (Feature Flag / Toggle / Switch) code to control the distribution logic, generally does not require complex tools and smart release with LB, it is a relatively low cost and simple distributed. This approach is supported by modern DevOps philosophy, research and development have the flexibility to customize and complete the self-publishing. The principle of switching function as shown below:

Function switch released images from Appendix 6.2

 Practice points

1. Release function switch to configure such a service requires a support center or switching centers, such as Ctrip Apollo distribution center in Appendix 6.3 or Appendix 6.4 FF4J open source, these switches support the release of the industry there is a special function switch SaaS services, such as LaunchDarkly Appendix 6.5. By configuring the center, or a research operation and maintenance personnel can dynamically switch the function configuration values ​​at runtime. Of course, the function switch announce a just distribution center usage scenario, distribution center also supports many other dynamic configuration scenarios.

2. Function switch services in general provide the client SDK, developers easy integration. At runtime, the client SDK latest synchronized switching value, technology for push mode (Push), also the pull mode (pull), or a combined push-pull mode.

3. New features (V2 new feature) and the old function (V1 old feature) living in the same set of code, the new features hidden behind the switch, if the switch is not open, take the old code logic, if the switch is open, take the new code logic. The technology can be understood as a simple if / else logic.

4. After applying on-line, switch to not open, then the operation and maintenance or research and development staff to open a new function by switching center, the flow through the verification of new features is no problem, then released complete; if there are problems, you can always switch back to the old function logic by switching center.

 Advantages and Applications

Advantage:

• Upgrade fallback switching and very fast

• With respect to complex publishing tools, implementation is relatively simple, relatively low cost

• R & D flexibility to custom publishing logic, self-publishing support DevOps

insufficient:

• Switching the whole amount, if V2 version has a problem, then have a direct impact on the user experience;

• There invade the code, the code logic becomes complex and require regular cleaning of the old version logic, high maintenance costs

Applications:

• There is a certain tolerance of the user experience scene

• Existing distribution center or service center switch

• It will not have the ability to develop complex publishing tools;

 Traffic patterns

A complete flow switching by the switching function, images from Appendix 6.1

3.2 A / B test

A / B testing Appendix 7.10 than the original product is mainly used for functional test, collect user feedback and comparative data product features designed to make decisions. In fact, A / B testing can also function as a new publishing technology. The following figure shows the realization is based on a A LB / B testing release.

 Practice points

1. The figure above, the original PC and the mobile terminal have access to the old version V1 services (also known as group A or control group), V2 When a new version (also known as B group or the experimental group) published, in order to verify the functional correctness of V2, but also in order when V2 have a problem to avoid affecting all users, first by LB traffic is switched end of the phone to the V2 version, after a period of a / B ratio for testing and observation (mainly through user and monitor feedback) to ensure that V2 is normal, LB by all traffic is switched to V2.

2. Implement A / B testing on the way LB, LB need to be able to do traffic routed through certain conditions, such as by client ip, device type, browser type, or even a custom HTTP Header or query string.

3. Advanced A / B testing requires specialized platform support, wasabi Appendix 6.6 is an open source intuit a senior A / B testing platform support, these platforms can be fine-grained to do A / B testing for certain types of users, such as for a user areas, users of a certain age, internal users, and so on. Gave the example, the assumption that a business-critical new feature line, in order to reduce the risk of the use of A / B testing can be done to only allow internal staff access to new features, to be the new functionality verified, then the whole amount of release to external users use.

4. Function switches and A / B test is somewhat similar, but the function is generally non-switching state and the total amount, can not be tested for a particular class of users, and the A / B test is typically a state capable of tracking transactions and user level state can be achieved tested to specific classes of users.

 Advantages and Applications

Advantage:

• User experience little impact;

• You can use the production flow test;

• It can be done to test for certain types of specific target user;

insufficient:

• Build a relatively high degree of complexity, there are certain technical barriers

Applications:

• The key core business, such as financial implications

• Have some A / B testing platform for R & D

 Traffic patterns

An A / B test for certain types of target audience, images from Appendix 6.1

3.3 Test shadow

For some upgrading legacy systems involved in the core business, to be sure, there is a big move called shadow testing, the use of more complex flow copy, playback and matching technology. Here is a sample architecture diagram shadow test

 Practice points

1. Target to achieve the old legacy migration services to upgrade to a new experimental service.

2. Before the test began, the need to deploy a legacy services and experimental service in a test environment, while two copies of the production database to the test environment. At the same time you need to request the production of logs collected, generally collected by kafka queue, and then through tools like goreplay Appendix 6.8, consumer request logs kafka inside, copying playback, distribute requests to legacy services and experimental service, receives a response for comparison, if all responses than successful, it can be considered legacy services and experimental services are equivalent in function logic; if there is a response comparison fails, the two are not considered functionally equivalent logic, we need to repair experimental shadow and re-test until all comparison is successful. The system complexity and criticality of different, shorter than the time of the test may take several weeks, long up to six months.

3. Because the bypass shadow test carried out in an independent test environment can be completely without impact on production flow.

4. Shadow test is generally applicable to equivalent reconstruct migration of legacy systems, such as .net turn Java, or SQLServer database upgrade to the MySQL database, and can not rely on outside too much, otherwise need to develop a lot of mock, testing deployment costs will be high, and the ratio the test is more complex and unstable.

5. Dangdang has a successful trading system turn .NET Java migration project appendix 6.9, using the shadow test technique, it is also useful reference.

 Advantages and Applications

Advantage:

• Absolutely no effect on the production of user experience

• You can use real production flow test (copy alignment)

insufficient:

• Build high complexity, high technical threshold, exported copy of the database is the difficulty

• External dependencies can not be too much, otherwise the test deployment costs are high, and the ratio of the test is more complex and unstable

Applications:

• The key core business, such as financial implications

• Have a certain shadow test platform for R & D capabilities, including traffic copy, reproduce and distribute the database export ratio of the system.

 Traffic patterns

Shadow test no impact on production flow, images from Appendix 6.1

Fourth, compare

The following table compares the various release strategy, a comprehensive comparison from various dimensions, for reference:

V. Conclusions and recommendations

Here are some suggestions for the selection of the release strategy for different stages of the company's reference:

1. Brute force release is generally not recommended, unless the development and testing environment, the user experience is not sensitive to non-critical applications, or business are short of time upset.

2. If you have not yet developed a more sophisticated rolling publishing tools and supporting intelligent LB, the function switch is a good cost-lightweight publishing technology, investment is relatively small, allowing developers the flexibility to customize issue logic.

3. Canary publishing receiving production traffic through a small new version of the server to verify that the new version, you can significantly reduce the risk. Canary issued for most scenarios, the general growth companies can be used.

4. For the body reaches a certain amount of business of the company, taking into account the user experience is critical to the business, we need to invest R & D resources to develop tools to support rolling release and supporting intelligent LB, release automation and achieve zero downtime. Rolling release and general release with the canary, starting a canary to verify the flow, then the incremental batch release.

5. With the lightweight virtualization (eg container) popular, two-server publishing group and has a faster release rollback rate, is worth putting the advanced publishing technology. Blue, green and deployment only for dual server group, rolling release can either be implemented on a single server group, can also be implemented on a two-server group.

6. For the new features involve critical core business lines, using A / B testing, can significantly reduce the risk of publishing, A / B testing is the only support for production testing for a specific group of users advanced publishing technology. Of course, A / B testing of investment is not low, it is recommended the use of certain research and development capabilities of the organization.

7. For critical core business of reconstruction of migration, to be sure, a big move last test is the shadow, the shadow test completely without impact on production flow and user. Of course, this big move and threshold input costs are high, it is recommended body mass organization put in enough business and R & D capabilities.

8. The above-mentioned various publishing strategy is not either-or, a company will often use a variety of integrated publishing technologies as complementary, flexible publishing capabilities. Release means is the mainstream e.g. canarypox + rolling release, certain lines of business may be required release switch function according to service scenario, there are some lines of business may employ advanced A / B testing release means.

VI Appendix

1. https://github.com/ContainerSolutions/k8s-deployment-strategies

2. https://opensource.com/article/18/2/feature-flags-ring-deployment-model

3. https://github.com/ctripcorp/apollo

4. http://www.ff4j.org/

5. https://launchdarkly.com/

6. https://github.com/intuit/wasabi

7. https://blog.zenika.com/2017/04/19/migration-dun-legacy-avec-goreplay/

8. https://github.com/buger/goreplay

9. http://blog.shurenyun.com/untitled-9/

10. https://en.wikipedia.org/wiki/A/B_testing

2、

2. Return to top

 

3. Back to top

 

4. Top

 

5. Top

1、

https://www.cnblogs.com/apanly/p/8784096.html

2、

 

6. Back to top

 

Author: ylbtech Source: http://ylbtech.cnblogs.com/ This article belongs to the author and blog Park total, welcome to reprint, but without the author's consent declared by this section must be retained, and given the original connection in the apparent position of the article page, otherwise We reserve the right to pursue legal responsibilities.