In the previous article "The Road to DevOps Transformation Success - The Meaning of Transformation and Five Misunderstandings", we started from the analogy between the bird flying school and the aerodynamic school. The transformation of DevOps cannot copy the implementation process of other organizations, but should go deep Understand the principles, principles and practices behind it. The previous article focused on five misunderstandings of DevOps transformation, namely: abandoning existing personnel and recruiting new DevOps experts, conducting large-scale organizational restructuring, rewriting applications and moving to the cloud, purchasing a package of DevOps tools, and developing a production environment Full access.
So what should be the correct posture for DevOps transformation? This article will focus on describing the top five practices of DevOps transformation, as well as specific implementation recommendations for the transformation.
Five Practices for DevOps Transformation
Practice 1: Get used to working in small batches (development, architecture, evolution of organizational culture)
Lasting change needs to be done in small batches and continuously, through repeated experiments and rapid learning based on feedback loops to find the most correct implementation path. This requires breaking down big problems into a series of small problems and solving them incrementally, maybe not as exciting as Big Bang-style changes, but this is the right posture for us to succeed.
1. Find the most important job
The most classic example is project management, traditionally planning and requesting budgets on a semi-annual or annual basis, which drives us to work on large and complex projects, where a lot of time is spent on features in a backlog waiting to be analyzed, estimated, and approved and prioritization and other transactional work. A white paper called "Black Swan Farm" shows that they analyzed 3,000 backlogs of different requirements waiting to be developed, using a prioritization decision-making method based on the cost of delay (the cost of lost weekly if this requirement is not done).
They found that there were three features on the backlog, each of which cost the organization $2 million per week if not delivered. These few features are hidden in a huge to-do list, but they are really critical. They realized that they should stop all work and deliver these three features as quickly as possible. Statistically, this situation follows a power-law distribution, as shown in the figure below.
So our job is to find the ones that really matter among the multiple projects, and that requires more transparency, clearer priorities, and collaboration from everyone in the organization.
2. Continuous evolution of the architecture
It is very common that many organizations have a large number of legacy systems, some software or hardware is too old, maybe the manufacturer no longer supports it, and some organizations do not even have source code for individual systems (which may be in the hands of Party B or have been lost). These types of organizations usually choose to solve the problem through a re-architecture of up to a year or two, and when it reaches a year, they may say that the complexity of the system is too high, and we need an additional two years! I have personally experienced such a large-scale project, and the CTO in charge of the final change has not been completed.
The key problem with the scenarios described above is that the focus is often on the technical aspects of the architecture rather than the results that need to be achieved . Survey data shows that if you answered yes to the following questions, then you are more likely to do continuous delivery and DevOps well and become a high-performing IT organization:
- Can you make extensive design changes to your own system without the authorization of someone outside your team or another team?
- Can you do your job without fine-grained communication with others outside your team?
- Can it deploy and release its own product or service on demand, independently of other dependent services or products?
- Can you test most of your own systems on demand without relying on a complex integration test environment?
- Is it possible to perform a non-disruptive deployment during normal business hours?
You can achieve all of the above effects on old legacy systems, but you may not be able to achieve any of the above effects in the case of high technology and new technology. So we want to focus on the results of architectural evolution, not just using cool technology .
More about the evolutionary architecture, as well as the idea of the strangler mode, I also introduced in my previous article, the main principles are as follows:
- Start with delivering the new functionality required by the business (at least initially) using Service Oriented Design
- Don't rewrite existing functionality unless you can keep it simple
- Faster delivery through continuous splitting
- Design for Testability and Deployability
- The new architecture runs on a PaaS platform
3. Continuous evolution of organizational culture
Organizational and cultural changes are also applied using continuous evolution. There are a variety of employees in an organization, some people are very interested in new methods and technologies, and some people are conservative and reluctant to try to make changes.
You can’t make a one-size-fits-all change across your organization, you should start with the people who most agree with DevOps philosophies, methods, and techniques . Accepting a new idea requires crossing the chasm. We first find a team that agrees with the principles and practices of DevOps, focus on the group with a certain risk tolerance, and quickly achieve the benchmarking effect of transformation, laying a solid foundation and giving people confidence. We don't need to spend a lot of time converting conservative people early on, and the most important thing is to get the first shot.
Practice 2: Create a feedback loop
On the basis of working in small batches, we want to establish a feedback loop. Feedback loops allow for continuous learning and continuous improvement based on learning , which are key principles of agile and learning organizations.
The continuous delivery pipeline is the specific implementation of the feedback loop, which can provide feedback information of multiple levels and links, and can achieve a balance between feedback efficiency and feedback integrity .
The following are the design drawings and renderings of the full open source continuous delivery pipeline that I cooperated with my friends last year, which fully reflects the principle of successive upgrades and feedback loops of the pipeline.
All open source pipelines can only meet the needs of small and medium-sized enterprises. The design and implementation of pipelines in large enterprises are more complicated. I will introduce them to you later when I find time.
Practice 3: Cross-functional collaboration through value streams
The roles of requirements, development, testing, information security, and operations need to work together in the software delivery value stream. A value stream map is a key tool for fostering cross-functional collaboration . We can identify the various roles that support the value stream and the collaborative relationship between the roles through the workshop for value stream sorting. Instead of documenting the minutiae of each step in the value stream, we understand how value is delivered and how roles work together.
In addition, the quality control part of cross-functional collaboration should be emphasized here. Not just automated testing, but also focus on continuous testing, continuous security checks, and make these activities repeated in the daily work to achieve built-in quality.
Practice 4: Build an organizational culture for experimentation
The survey shows that organizational culture can be measured, and organizational culture has a measurable impact on IT effectiveness and organizational performance. We use the Westrum model to measure organizational culture. There are three types in this model:
- Pathological organizations are characterized by an abundance of fears and threats. People often hide or suppress information for political reasons, or even distort facts to make themselves appear better.
- Bureaucratic organizations are characterized by the self-preservation of each department. Each department wants to maintain its own 1/3 of an acre, stick to its own rules, and usually follow its own bylaws.
- Generatives focus on their mission and how to get there. Anything has to give way to high performance, a high degree of teamwork, risk sharing, and encouragement of bonding. Facing failure requires trying to find the root cause of the problem in the system, not scapegoating or blaming each other.
According to the statistical results, a high-trust bio-type culture is not only very important for creating a safe working environment, but also the foundation for building a high-performance organization.
The above models not only stay at the theoretical level, but can also be applied in the practical field.
Case 1. Google's disaster recovery test
Google conducted a study a few years ago on how to build a good team. They surveyed 200 respondents from 180 Google teams, looking at over 250 different factors, such as someone on the team with a PhD, someone with an extrovert, someone obsessed with AngularJS, etc. But they ultimately found that the number one factor in building a high-performing IT organization was the psychological safety of being able to take risks in a team without feeling unsafe or hurt.
We need to build a safe environment where failure is acceptable and serves as a foundation for organizational learning . Both Google and Amazon conduct disaster recovery testing online to ensure that in the event of a critical failure, the failure recovery strategy really works and the system remains functional.
Google has an entire team dedicated to planning and conducting disaster recovery tests, and even for a year simulating an alien invasion of Silicon Valley where they disconnected Mountain View from the outside world, shut down data centers, and had a real impact on the infrastructure, but The team is required to maintain service level objectives. The disaster recovery tests they designed require the collaboration of engineers from many different groups who usually do not work together, so when a large-scale disaster really hits, they have established a strong working relationship.
Case 2. Etsy's "Three Sleeve Sweater" Reward
Etsy hosts an annual developer conference, and rewards the maker of the biggest disruption to the production environment over the past year with a "three-sleeve sweater". So why is the reward a three-sleeve sweater? Because there's a picture of a three-sleeve sweater on Etsy's 404 page.
The classmate in the sweater in the picture is Ryn, the maker of Etsy's biggest failure last year. She recorded the failure process on her blog, including when and why the production environment was interrupted. When a breakdown occurs, she immediately asks for help on Slack, and gets an immediate response from a lot of people around her, who then swarm together to quickly fix the problem. Afterwards, they held a post-mortem failure analysis meeting that was exempt from liability, and gave specific measures to prevent further failures and optimize the system . Ryn also received last year's award for promoting organizational learning and making systems more resilient.
Case 3. Netflix's Chaos Monkey and the Simian army
This tool from Netflix continues to wreak havoc on production environments, verifies that systems are well-resilient, and helps engineers build more robust systems.
Practice 5: Continuously eliminate waste and optimize the value stream
DevOps requires continuous improvement, removal of waste, and a smoother flow of value. I recently coached some teams to use value stream mapping to sort out processes and problems, and I found that it is really helpful for organizations to understand and optimize existing software delivery processes.
We can invite leaders from different departments such as product, development, testing, operation and maintenance, security, etc. to participate in the value stream grooming activity. In fact, the hardest part is to get these people together in a conference room at the same time. We comb through the process from requirement proposal to coding, test verification to deployment and release one by one. During the process, we will find that everyone's cognition is inconsistent, especially for lead time, waiting time and C/A% (complete and accurate ratio) 's estimate.
It is important for everyone to achieve an understanding and awareness of the entire value stream, but it is even more important to identify specific measures and expectations for how to improve in the future . Periodic (e.g. monthly) reviews and continuous improvement based on the different roles agreeing on goals. In the process of improvement, it is not to tell the team how to carry out the work in detail, but to clarify the goal and let the team deeply participate, think independently, and find ways to achieve the goal through continuous experimentation and learning. (This mirrors the misunderstanding we mentioned earlier)
Specific Implementation Recommendations for DevOps Transformation
In the past five years of research on high-performance enterprises, the key competency elements of DevOps transformation have been summarized , as shown in the following figure. There are four areas in the figure, namely software development practice, lean product development, lean management, and change leadership, and more than 20 competencies have been refined in these areas. The construction of these capability items can be used as the main reference system for the implementation of DevOps transformation, and it is strongly recommended that you continue to pay attention.
The transformation of DevOps is not simple. If you want to embark on the road to success, here are a few Tips:
- Agree on measurable business goals . Set "one-hop-at-a-hop" goals, such as reducing the number of critical bugs by 10%, increasing service availability by 20%, doubling the release frequency, or other mixed outcome metrics.
- Give team resources to experiment and motivate learning . The team cannot "work overtime" to explore improvements under the premise that the daily work remains the same. Improvements require real effort and should be prioritized in the same way as developing new features and fixing bugs. Specifically, you can use Kanban to manage WIP, assign dedicated teams to work on improvements full-time, or give teams a specific time each week to work on improvements.
- Communicate with other teams to encourage collaboration . Teams like compliance, security, governance, etc. are often cited by many people as barriers to improvement. But in fact, if you have a friendly conversation with these teams, you may find that they are very eager to discuss specific measures to achieve a win-win situation.
- Get quick wins . You need one to two months to make improvements. Specifically, there are three key points: first, through the value stream map or constraint theory, find a problem that has the greatest impact, and can be quickly solved and measurable; second, limit the scope of the first improvement and minimize the improvement work; Then, work with a team that is interested in improvement and has sufficient capacity and capability for quick wins.
- Share successful experiences . In order to gain the support of others, you need to do a good job of publicizing the successful experience and attract more people to join the improvement work. For example, the DevOpsDays conference within some enterprise organizations, which promotes experience across departments, is very effective. Also lunch-and-learn sessions, internal blogs, mailing lists, Slack or HipChat channels are some of the best ways to get participants. Also help others who are trying, as a DevOps coach should do.
- Continuous improvement . High-performing organizations always seek opportunities for improvement. As Taiichi Ohno, the founder of the Toyota management system, said: "Kaizen [improvement] opportunities are infinite", the opportunities for improvement are endless. Lu Qi, President and COO of Baidu Group, also said in a speech last year: "Engineering Excellence is a never-ending, personal, team, ability pursuit and tool platform innovation. long-term, core competitiveness". Relentless pursuit of excellence, this is the attitude we should adhere to .
Summarize
- Five practices for DevOps transformation . Getting used to working in small batches (development, architecture, evolution of organizational culture), creating feedback loops (pipelining), cross-functional collaboration through value streams, establishing an organizational culture of experimentation, continuously eliminating waste and optimizing value streams;
- Specific implementation recommendations for DevOps transformation . Focus on key competency elements of DevOps transformation, agree on measurable business goals, give team resources to experiment and motivate learning, communicate with other teams and encourage collaboration, achieve quick wins, share successes, and continuously improve.
The above contents are summed up in many enterprises and are the most effective ways to improve organizational performance that have been proven. We aim for faster releases, higher reliability, better resilience and safer systems, and a more humane, continuous improvement and self-reinforcing organization.
I hope this article can provide a little light on your DevOps transformation, and I wish you an early start to DevOps success!
DevOpsDays Conference Beijing Station Registration Channel
On May 5, 2018, have a face-to-face chat with the great god Jez Humble about DevOps!
It is a rare opportunity to meet and communicate with the Great God, hurry up to scan the code to register!
