Unless you've been in hibernation for the past year or two, I'm sure you're well aware of the benefits of continuous testing: lower development costs, less waste, improved system reliability, lower release risk, etc. Of course, you think that in Putting this into practice in the real world is not as simple as it is with some vendors. Obviously, having a continuous integration tool in place doesn't mean the team has successfully implemented continuous testing.
Fortunately, we now have more resources than ever to overcome this organizational challenge, such as the ACT framework. Not long ago, my team and I also released our "Ultimate Guide to Continuous Testing", which is very much in line with the philosophy of the ACT framework. In this post, I'll share some key considerations and tips for a successful continuous testing implementation in my guide.
Assess your current team, technology, and process
Anytime you want to improve software development outcomes, you must consider three fundamental, closely linked pillars: process, tools, and people. Teams can continuously learn and improve by improving the process. Still, the tools they use and the team members themselves must adapt in order for the new and improved process to be successful in the long run.
Here are a few questions to ask:
Do we have a common understanding of our delivery pipeline? Don't just think about the development pipeline, think about how to create value for your business.
How much of it is automated? Do we have the right CI/CD engine?
Are our current testing tools and processes integrated with the above pipeline?
Are we missing the expertise needed for automation? Should we invest in training? If so, where?
What are our biggest constraints: time, budget, resources? How can we counteract these?
Knowing which quality factors are most important
For every team, some quality factors will be more important than others, which is why it's not always necessary or wise to focus on continuous testing of every factor of software quality. For example, if you are developing a system for a very diverse group of users, usability testing may take precedence over performance testing. For example, a government system that citizens use to file their taxes may require more availability and security than performance.
According to the ISO 25010 software product quality standard, there are eight software quality attributes: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. For each, you have a corresponding set of tests to assess its level of quality.
Work with your team to assess the areas of greatest concern to users and stakeholders regarding software quality. What's the worst case scenario that would cause the most damage if something broke? These are risks that must be addressed first. With limited testing time between releases, which could be weeks, days or even hours, following a risk-based approach will help clarify which ones are worth prioritizing.
Planning Your Continuous Testing Pathway
There are several quality areas to focus on that will help you in your efforts to achieve an efficient continuous testing program, including shift-left and shift-right testing.
In the diagram below describing the three testing maturity levels (Basic, Efficient, and Continuous), you can see how different aspects of quality from source code, environment, test automation, etc. can be managed to pull off continuous testing. For example, before deploying CI/CD, you must first implement source code version control.
I encourage you to use this chart as a reference to see where your team is in the nine different quality areas. Go through the lines from top to bottom, thinking with your team members whether you approach each given area in a way that is characterized by basic, productive, or continuous testing.
From there, you can develop an action plan to progress through those maturity levels, enabling continuous testing. They address which areas need improvement first, the most critical areas to support the next step of continuous testing, and the software quality attributes your team has identified as the most critical.
Planning for Success
As the saying goes, "If you don't plan, you plan to fail." Before aggressively engaging in continuous testing and automating everything that can be automated, it's wise to first figure out what success should look like based on your team's context and business goals. It's okay if your team makes some mistakes along the way, and your continuous testing is a well-oiled machine, it doesn't happen overnight!
Remember, it starts with understanding your organization's goals, people, processes, and tools. Understand the most important risks mitigated through continuous testing and develop a concrete, realistic action plan accordingly.