Modern Software Engineering—Part 2: Testing

If debugging is the process of removing software defects, then programming must be the act of putting them in." - Edsger Djikstra

Writing automated software tests is like playing a game of telephone with yourself—you're the only one to blame when you misunderstand what the information is. This is hard enough if you're writing tests for your own code, but consider that you're writing tests for code written by someone else, which is never tested in the first place. Now, it's like trying to understand what the message is on a piece of paper that's been washed three times in a blue jeans pocket! This is a test written for!

This is a test written after the code being tested has been written. Now think about writing tests first - it's like playing master's game against yourself by writing some spec or test that makes sense first to make sure that the solution you're about to write will "do the right thing". But if you studied computer science, this sounds a lot like solving the stalling problem — only worse, because now you not only need to prove to yourself, but also to the compiler/interpreter, that what you want it to do is correct things.

So why, for the past 20 years, testing has been an integral part of modern software engineering practice — whether testing first or testing last, we professional software engineers still need to think about how to test and validate software to meet requirements.

IntroductionIntroduction

It’s story time again.

Last time in Part 1 , I wrote about how I learned how to design systems to accommodate modern requirements for scalability, reliability, availability, maintainability, and security. Designing a solution can only go so far, because at the end of the day, the solution needs to be implemented—and sometimes, it has to be done by a team or multiple teams.

As you can imagine, coordinating work across teams will be a major source of problems, but is there anything we can do to reduce this burden? Here comes automated testing—especially the kind that specifies behavior rather than testing implementation.

When I worked at Friendster, I knew exactly what the customers of the services I was doing expected. However, this is not completely specified - we have a protocol that we follow (this is before protocol buffers were popular) and some URIs that are called by these clients. The semantics are not fully spelled out, but we have a loyal audience — I can read the client's code and figure out what to expect from the current implementation.

This is important - instead of creating a completely new protocol or creating a new contract, we start with known requirements, which we can write as automated tests. One of the first few things I did was turn these tests into specifications that I could program and gradually bring the implementation to the point where it met the requirements. This work resulted in two products:

• C++ Network Library — A reasonably performant C++ implementation of an HTTP client and server that the service I'm rewriting will be integrated on top of.
• memcache++ library — A reasonably performant C++ implementation of memcache client that supports sharding and virtual node pools.

Both open source solutions are the result of internally defined technical requirements. We start with an existing system, break it down into its component parts, and gradually implement the solutions until we can share the non-business-critical parts as open source software.

You may ask, why do I need to start with testing? Because testing allows me to fill in solutions to meet needs in an incremental and predictable way. With tests, I and the people who review the code I write can understand what the requirements are and automatically verify them by running the tests. This gave us the confidence we needed and we got a solution that met our needs.

Having tests in place allows me to focus on tests that provide necessary and sufficient functionality while giving me the confidence to refactor and improve the solution and quickly verify that I'm not breaking coding requirements. By having tests cover requirements, I've been able to catch many bugs and deliver functionality quickly, fearlessly refactoring along the way.

This was around 2007–2008, and a lot of these concepts like test-driven development and behavior-driven development were just starting to become popular, but generally in the enterprise software industry. Here I take some of these good ideas and apply them to microservices and horizontally scalable systems!

Fast forward a few years and we're now in 2023, testing has become a dirty word in some circles (TDD and BDD tend to burn out a lot of people, mostly due to a misunderstanding of the principles), and has become a As an afterthought, we asked our co-pilot to do unit tests for the code we wrote. This is a bit of a shame, because the freedom that high-performing software engineering teams employ the right types of testing to adapt to changing requirements and improve the implementation of solutions is extremely valuable, and those teams that don't invest early often realize too late that Testing could have saved them major failures, sleepless nights due to bugs creeping into production, or simply losing business due to poor quality and speed of the solution.

In this article, I will write more about the role of testing in modern software engineering and how doing it correctly will enable you and your team to be successful in this industry.

Testing Levels

Before we go any further, it’s a good idea to understand the different levels or categories of testing. If you haven't written tests before, it might be beneficial to know that there's a fairly robust taxonomy of testing terms so that you can at least keep up with the discussion happening around them.

• User Acceptance Testing (UAT)—usually automated testing that ensures that a software system meets end-user requirements. Simulating an end user typically involves driving the software's user interface (automated testing of browser-based web user interfaces, application drivers for native application user interfaces, API service clients, etc.) to do what the user would do and observe The result of these actions is to see if it meets the software's acceptance criteria. This is usually the highest level of testing and covers everything from the entire software system.
• System Testing — Automated testing that typically tests the functional and non-functional attributes of a system. Here, a system can be a fully integrated application or a subsystem with related components. System tests are usually more comprehensive than user acceptance tests (UATs).
• Integration Testing—Typically automated testing that tests the interaction of multiple components working as subsystems in an integrated environment (usually a test harness or application rack), which facilitates the routing and testing of integrated components. Integration testing typically tests the logical subsystems of a complete solution that work together to provide a specific set of functionality.
• Unit Testing - Typical automated testing that tests the functionality of a single component (not necessarily a single class) in isolation. Dependencies can be replaced by functionally equivalent implementations in order to simulate these dependencies in a controlled (sometimes intentional) way.

In some cases, you may encounter the need to have manual or human-driven testing to cover some unpredictable or combinatorially large possibility space (think computer games, artificial intelligence models, control systems, etc.). These still have a good place in the software engineering industry, but in this article I will focus on automated test cases.

Now that we have some definitions, let’s dive into some modern software engineering testing methods and how it’s changing the way we solve problems.

Test-Driven-Development (aka TDD)

Test Driven Development or TDD is a methodology for implementing software by first writing the tests (or specifications) into executable code, seeing the tests fail (first in red), implementing a solution to meet the requirement, seeing the tests succeed (green), Refactor solutions for readability and flexibility while keeping tests running successfully (staying green) and iterating. Here's more explanation of each step in this methodology:

1. Write a failing test to represent a requirement . This might use a class that doesn't exist yet, or a method that hasn't been implemented yet, or a situation that hasn't been handled by an existing implementation, or some new behavior that the system hasn't implemented yet — whatever the new requirement is, write a test to represent the requirement as something executable and initially fail. This step lets us think about the missing functionality and how it is used at any level - the tests could be UAT, system tests, integration tests or unit tests.
2. Implement solutions to meet needs (green) . An initial implementation that satisfies the requirement is probably the easiest job, or a simple "return something the test expects" (I know, it feels like cheating, but trust the process...), just so you can see the test" Go green”. This step forces us to think about the most direct way to solve the problem so that we can move to the next step and do the cycle again.
3. Refactor ruthlessly while keeping your tests green (keep them green) . Don't pass step 2 here, because the meat of software engineering happens at this step, here we can take a look at the interfaces used in testing and implementation to see if we are getting closer to a more maintainable and flexible solution, or are we Is more testing needed to find the pattern. The more tests you have that cover the functionality of the system, the more you will need to refactor, not just the implementation, but also the tests — if you also follow domain-driven design, this allows you to refine the model in the system so that you have a better understanding of the solution Your understanding will continue to evolve as the model changes.
4. Iterate . As you cover more and more functional and non-functional requirements of the system at different levels (UATs, system tests, integration tests, unit tests), you will inevitably find that some requirements are no longer requirements, existing requirements Slight changes occur due to new business requirements, and you may have to start over from certain subsystems. Recognizing when to add a test, remove a test, optimize performance or efficiency, or just call it done and move on is an important part of the process. As long as you're not done yet, go back to step 1.

Note that you can start following TDD even if you already have a code base without tests. You can go top-down (from UATs to unit tests) or bottom-up (from unit tests to UATs) and along the way start refactoring your interfaces so that you feel more confident that they represent the logical components or domain model .

There are many benefits to following TDD from the beginning:

• You have to consider requirements and design when writing tests . Code that is difficult to test usually means that it does not follow good design practices. If you find that you cannot express the test well, it means that you do not understand the requirements well, which forces you to first understand what the requirements are before writing tests.
• You have more confidence that the solution you have will meet requirements even before it reaches production. Catching problems in advance can save you from foreseeable troubles in production. It also allows you to focus on solving coaching problems that already exist, rather than waiting too long to find out whether you've met a need. Instead of spending time debugging, spend your time solving other problems and confidently delivering value incrementally.
• You have time to organize and make it beautiful . TDD explicitly makes the time for refactoring a part of the development process—rather than deferring it to a later date. If you are pressed for time and need to postpone refactoring, it can also be done later, since your tests represent the state of the requirements and the quality of the implementation can be improved as part of the process.

That's fine too, but we also need to acknowledge the costs and some drawbacks of TDD.

• Writing and running automated tests isn't cheap . Some kinds of tests are more difficult to write than others, and not all of them produce the same value. Writing UATs may require expertise in a specific testing framework or, when simulating production deployments, the use of special hardware that is not readily available (such as a powerful GPU or FPGA). Some require a full deployment of multiple services, which may not be cost-effective to set up for testing purposes, so there may be some detours.
• It's hard to show the value of testing code, especially when it's seen as an opportunity cost . A lot of people still think testing is a waste of time because the important thing is shipping things that work — if they fail in production, we hack it because we need to make enough money so the funds don't run out. Unfortunately, shipping code to production instead of testing means you're risking the effectiveness of your solution and business every time you deploy code and new functionality. Although TDD is a good practice, and there are plenty of success stories showing why following TDD is a good idea, unless effective test coverage is viewed as an insurance policy against future failures and requirements changes, then it will be a pain point .
• When you confuse effective testing with 100% test coverage, you're going to have a bad time . TDD is not about achieving 100% test coverage, but rather focusing on expressing requirements into executable tests. You can have as many tests as you want, as long as they represent important aspects of the system you're building. Having 100% test coverage does not indicate how effective your tests are at representing important aspects of the solution. It's probably that a smaller test set gets the best value in making sure the problem you're trying to solve is solved.

TDD is not a panacea for all software quality problems we encounter in the world. However, it is a practice that can help maintain important focus so that we can confidently design systems that meet our needs.

Automated TestingAutomated Testing

If you already follow TDD, that's fine. But if you don't, it's still important that you have tests that can run automatically when:

• While developing, in the "internal developer loop". If you can't run tests in your integrated development environment or on your workstation to quickly verify that your solutions are doing what they're supposed to do, you're going to have a bad time. Ensuring that automated tests can be built/run quickly and represent important requirements is a significant productivity booster and worth the investment. If you do more than write automated tests that developers can run on their workstations, you've achieved 80% of the benefits of automated testing.
• Maintain a regression test suite. Whenever a bug is raised or discovered, the first thing you should do is reproduce it with the failing tests. This way you can manage the process of fixing bugs as another requirement, expressing it as a test that catches regressions (meaning, the software won't exhibit a bug that has been fixed in the past). The more bugs you turn into regression tests, the more broadly you can express the actual requirements on the system and prevent them from recurring in the future.
• Also test the non-functional aspects of the system. Non-functional requirements refer to system qualities that are not strictly tied to functionality—such as throughput, latency, resource consumption, minimum load requirements, and other observable attributes. Automating these requirements allows you to make them part of the design and implementation requirements, so that when changes are made to the system, they are always taken into account.

Automated testing is becoming a key tool in delivering competitive and higher quality software systems, especially in the modern software engineering practices we see today. Given the complexity and criticality of the systems we are building and deploying, it's difficult to see how we can manage moving forward without automated testing.

Modern Testing Techniques

Assuming you've implemented automated testing, you have UATs, system tests, integration tests, and unit tests that you can run. You also have a regression test suite and non-functional requirements expressed as automated tests. How do you bring your testing practices into the modern era of software engineering?

Especially for software that is developed and deployed as a distributed system in the cloud, orchestrated in an environment like Kubernetes, where the control plane manages the placement and management of workloads and resources respectively, public cloud providers provide for network presence and geographic diversity. To provide management resources, the architecture of applications is becoming more and more complex. Testing these applications becomes very difficult and expensive.

Here are a few things to consider to manage this complexity and ensure you can keep up with the demands of modern large-scale, globally available services:

• Invest in continuous integration and continuous delivery . Testing your software before it goes into production will only get you so far, but the realities of production are rarely anticipated in development. Having a way to consistently ship code tested in an integration environment to production in a controlled and safe manner is key to being able to adapt your solution to the realities of production. Because you've invested in testing, bugs you find in production can be expressed as failed tests and automatically run through your continuous integration (CI) and continuous deployment (CD) pipelines. This reduces time to market and shortens feedback cycles for engineering teams.
• Invest in fuzz testing and automated fault finding . There are a plethora of solutions to automatically inject faults into the systems you depend on, be it remote API services or internal components. Fuzzing is a testing method that uses randomly generated inputs to find potential security holes or unexpected problems. While not a replacement for handwritten tests, these tests can augment the requirements-driven tests you write for your system to catch potential failures early in the development process.
• Small scale testing (also known as canary testing) in production gives the most realistic testing of your system. Make testing in production a critical link in your application delivery and deployment pipeline.
• Leverage artificial intelligence and LLMs to increase your test coverage. If you have access to GitHub Copilot or similar technology, consider using them (after consulting with legal counsel about the impact on your company and code) to fill in unit, integration, and system testing of your existing systems. Or, if you start using TDD, consider AI automation to reduce the time spent developing these tests in the inner loop. After all, if developer time spent coding and testing is an issue, AI should be a good way to reduce that cost. )

As systems become more complex because they are distributed and the scale of processing increases, automated testing will only become more important to ensure the quality and correctness of the various interactive systems we are developing.

Summarize

Writing and maintaining effective automated tests represents a critical requirement for software systems and is becoming a pursuit and important skill for today's software engineering professionals. Gone are the days of having testing experts, as now everyone is a developer and operations engineer. Modern software engineering requires that every software engineering practitioner knows and understands the value of automated testing and how it affects the robustness, quality, and effectiveness of the software systems we deliver to our customers.

At the end of the day, software engineering is about building the right thing to solve the right problem. Knowing what the requirements are to solve this problem is the key to being able to solve it effectively.

Thank you for reading!

 Modern Software Engineering—Part 1: System Design