10 Tips for Embedded Software Testing

In embedded software development, generally speaking, the ratio of time spent testing to coding is 3:1 (it may actually be more). This ratio continues to decline as your programming and testing level increases, but no matter what, software testing is very important to the average person.

Many years ago, in order to have a deeper understanding of embedded, a developer asked Oracle such a question: How can I know and understand what my system is doing? Oracle was a little surprised by this question , because no one asked that at the time, and most of the contemporary embedded developers asked superficial questions such as "how can I make the program run faster" and "what compiler is the best".

Therefore, facing this unusual but mature question, Oracle was delighted and replied seriously: Your question is very deep and mature, because only by constantly deepening your understanding can you continuously improve your level. And in order to encourage this persistent programmer, Oracle told him 10 tips about embedded software development and testing:

1 know how to use tools

Usually embedded systems have higher requirements on reliability. The failure of embedded system safety may lead to catastrophic consequences, even for non-safety systems, it will cause serious economic losses due to mass production. This requires rigorous testing, validation and validation of embedded systems, including embedded software. As more and more fields use software and microprocessors to control various embedded devices, it is increasingly important to quickly and efficiently test the increasingly complex embedded software.

Just like repairing a car requires tools, a good programmer should be able to use various software tools proficiently. Different tools have different scopes of use and different functions. Using these tools, you can see what your system is doing, what resources it takes up, and what external things it is dealing with. The problem that frustrates you for several days may be easily solved by a certain tool, but you just don't know it. So why do so many people always think of using testing tools after tossing half to death? There are many reasons, mainly two. One is fear, the other is inertia. The fear is because adding test tools or test modules to the code requires skill and may introduce new errors, so they always like to hope that the bugs can be eliminated by constantly modifying and recompiling the code, but the result is of no avail. Lazy because they are used to using simple means of testing like printf. Let's introduce some commonly used embedded testing tools.

1. Source-level debugger [Source-level Debugger]

This kind of debugger generally provides functions such as single-step or multi-step debugging, breakpoint setting, memory detection, and variable viewing, and is the most fundamental and effective debugging method for embedded debugging. For example, the gdb provided by VxWorks TornadoII belongs to this kind.

1. Simple and practical print display tool [printf]

printf or other similar printing and display tools are estimated to be the most flexible and simplest debugging tools. Printing various variables during code execution can let you know the status of code execution. However, printf interferes a lot with normal code execution (generally, printf takes up a long time on the CPU), so it needs to be used with caution. It is best to set the print switch to control printing.

1. ICE or JTAG debugger [In-circuit Emulator]

ICE is a device used to simulate the CPU core. It can detect the internal working conditions of the CPU in real time without interfering with the normal operation of the arithmetic unit. It also offers what desktop debugging software offers: complex conditional breakpoints, advanced real-time tracing, performance analysis, and port analysis. ICE generally has a relatively special CPU called a bond-out CPU. This is a CPU that has been packaged open, and through special connections, you can access the internal signals of the CPU, and these signals cannot be "seen" when the CPU is packaged. When combined with powerful debugging software on a workstation, ICE provides the most comprehensive debugging capabilities you will find. But ICE also has some disadvantages: it is expensive; it cannot work at full speed; similarly, not all CPUs can be used as external CPUs. From another perspective, these external CPUs are unlikely to be replaced by new CPUs in time. replace. JTAG (Joint Test Action Group) Although it was originally developed to monitor IC and circuit connections, this serial interface has expanded usage, including support for debugging. Visual Dsp++ designed by AD Company for Blackfin supports high-speed JTAG debugging.

1. ROM Monitor [ROM Monitor]

A ROM monitor is a small program that resides in the embedded system ROM and communicates with debugging software running on the workstation through a serial or network connection. It's a cheap way, and certainly the lowest-end technology. It does not require any special hardware other than a communication port and a small amount of memory space. And provides the following functions: download code, run control, breakpoint, single step, and observe, modify registers and memory. Because ROM Monitor is part of the operating software, it only works when your application is running. If you want to check the CPU and application status, you have to stop the application and enter the ROM monitor again.

1. Data Monitor [Data Monitor]

This kind of monitor can not only display the content of the specified variable without stopping the CPU, but also collect and display the change process of each variable in a graphical form.

1. OS Monitor [Operating System Monitor]

Operating system monitors can display events such as task switches, semaphore sending and receiving, interrupts, etc. On the one hand, these monitors can show you the relationship and time connection between events; on the other hand, they can also provide diagnostics for problems such as semaphore priority inversion, deadlock, and interrupt delay.

1. Performance analysis tool [Profiler]

It can be used to test where the CPU is consumed. The profiler tool can let you know where the bottleneck of the system is, the CPU usage and the places that need to be optimized.

8. Memory test tool [Memory Teseter]

You can find the problem of memory usage, such as memory leak, memory fragmentation, memory crash and so on. If you find that the system has some unpredictable or intermittent problems, you should use the memory test tool to test it.

9. Run the tracker [Execution Tracer]

It can display which functions are executed by the CPU, who is calling them, what are the parameters, when they are called, etc. This tool is mainly used for testing code logic, those that can find abnormalities in a large number of events.

10. Coverage tool [Coverage Tester]

It mainly shows which codes are specifically executed by the CPU, and lets you know which code branches have not been executed. This helps improve code quality and eliminate dead code.

11. GUI test tool [GUI Tester]

Many embedded applications interact with some form of graphical user interface, and some system performance tests are based on user input response time. The GUI test tool can be used as a script tool to run test cases in the development environment. Its functions include recording and playback of operations, capturing screen displays for later analysis and comparison, setting and managing the test process (Rational's robot and Mercury's Loadrunner tool is an outstanding representative). Many embedded devices do not have a GUI, but it is often possible to instrument embedded devices to run GUI test scripts. Although this method may require changes to the code under test, it saves time for functional testing and regression testing.

1. Homemade tool [Home-made tester]

In embedded applications, sometimes for specific purposes, it is necessary to write some tools to achieve certain testing purposes. The video stream recording and display tool I once wrote was of great help in testing the flow and changes of video conference data, and helped the company find several deeply hidden bugs.

2 Find memory problems early

Memory problems are very harmful and not easy to troubleshoot. There are three main types: memory leaks, memory fragments, and memory crashes. The attitude towards memory problems must be clear, that is, early detection and early "treatment". In software design, memory leaks have the greatest "fame", mainly because the continuously allocated memory cannot be released in time, and over time, the system's memory is exhausted. Even careful programming veterans sometimes encounter memory leaks. Friends who have tested memory leaks probably have a profound experience, that is, memory leaks are generally hidden deeply, and it is difficult to find them through code reading. Some memory leaks may even appear in libraries. It is possible that this is a bug in the library itself, or it may be that the programmer did not understand their interface documentation correctly and caused misuse.

Most of the time, most memory leaks go undetected, but may manifest as random failures. Programmers tend to blame this phenomenon on hardware problems. If the user's system stability is not very high, then restarting the system is not a big problem; however, if the user's system stability is very high, then this kind of failure may make the user lose confidence in the product, and it also means that your project is a problem. Failed project. Because memory leaks are so harmful, there are many tools to solve this problem. These tools find memory leaks by finding unreferenced or reused code blocks, garbage collection, library tracing, and other techniques. Each tool has advantages and disadvantages, but in general, it is better to use than not to use. In short, responsible developers should test for memory leaks to prevent problems before they happen.

Memory fragmentation hides deeper than memory leaks. With the continuous allocation and release of memory, large blocks of memory are continuously decomposed into small blocks of memory, thus forming fragments. Over time, when it is necessary to apply for a large block of memory, it may fail. If the system memory is large enough, the persistence time will be longer, but in the end, the allocation failure cannot be escaped. In systems using dynamic allocation, memory fragmentation occurs frequently. At present, the most effective way to solve this problem is to use tools to find out who is the culprit of memory fragmentation by displaying the memory usage in the system, and then improve the corresponding parts.

Due to various problems of dynamic memory management, in embedded applications, many companies simply disable malloc/free to avoid future troubles.

Memory crash is the most serious result of memory usage. The main reasons are array access out of bounds, writing of released memory, pointer calculation errors, access to stack address out of bounds, and so on. System failures caused by memory corruption are random and difficult to find, and currently there are few tools for troubleshooting.

In short, if you want to use memory management units, you must be careful and strictly abide by their usage rules, such as who allocates and who releases.

3 In-depth understanding of code optimization

When it comes to system stability, people think more about real-time and speed, because code efficiency is too important for embedded systems. Knowing how to optimize code is a must-have skill for every embedded software developer. Just like a girl loses weight, at least she knows where she needs to lose the most, so that she can buy weight-loss drugs or equipment to lose it. It can be seen that the premise of code optimization is to find the place that really needs to be optimized, and then prescribe the right medicine to optimize the corresponding part of the code. The aforementioned profile (performance analysis tool, some full-featured IDEs provide this built-in tool) can record various situations such as the CPU usage of each task, whether the priority of each task is properly assigned, and whether a certain data is copied. How many times, how many times the disk is accessed, whether the program for sending and receiving from the network is called, whether the test code is closed, etc.

However, profile tools are not enough in analyzing real-time system performance. On the one hand, people often use the profile tool after a problem occurs in the system, that is, the CPU is exhausted, and the profile tool itself occupies a large amount of CPU, so the profile may not work in this case. According to the Heisenberg effect, any testing method will more or less change the system operation, and this is also applicable to the profiler!

In short, the prerequisite for improving operating efficiency is that you must know what the CPU is doing and how it is doing.

4 Don’t Let Yourself Find a Needle in a Haystack

Finding a needle in a haystack is just a vivid metaphor for debugging.

I often hear people in the group say shit to the code they are debugging! It is understandable, because the code is not written by him, and he has enough reasons to shit the code full of bugs, as long as he does not write this kind of code himself, otherwise one day Others in the same group may also shit the code he wrote. Why is there a needle in a haystack? Someone must have dropped the needle into the sea; why did the needle fall into the sea? Someone must have been careless or sloppy. So when you are complaining that the needle is so hard to find, have you ever thought that you threw it away hastily? Similarly, when you are debugging half-dead, have you ever thought that you have to reflect on the fact that in order to seek shortcuts, you may not strictly abide by good coding design specifications, you may not check the correctness of some assumptions or algorithms, and you may not have some possible existence. How about marking the problematic code? For how to write high-quality code, please refer to Lin Rui's "High-Quality C++/C Programming Guide" or "0x8 "Books" About C".

If you do drop a needle in the ocean, you will have to take some precautions, such as wearing safety gloves, to prevent stabbing yourself before you find it. Similarly, in order to expose and capture the source of the problem as much as possible, we can design a more comprehensive error tracking code. How to do it? Handle the failure of each function call as much as possible, detect the validity of the input and output of each parameter as much as possible, including pointers, and detect whether a certain process is called too much or too little. Error tracking lets you know roughly where you dropped the needle.

5 Reproduce and isolate the problem

If you drop the needle not in the ocean but in the haystack, that's easy to write about. Because at least we can divide the haystack into many pieces and find them one by one. For large projects with independent modules, using the isolation method is often the last method to deal with those hidden deep bugs. If the problem occurs intermittently, it is necessary for us to try to reproduce it and record the entire process of making it reproduce so that we can use these conditions to reproduce the problem next time. If you are confident that the problem can be reproduced using the conditions documented, then we can proceed to isolate the problem. How to isolate? We can use #ifdef to close some code that may not be related to the problem, and minimize the system to the point where the problem can still be reproduced. If you still cannot locate the problem, it is necessary to open the "Toolbox". You can try to use ICE or data monitor to check the change of a suspicious variable; you can use the trace tool to get the function call including the passing of parameters; check whether the memory is corrupted and the stack overflow.

6 Retreat to advance

In order not to get lost in the forest, a hunter often leaves some marks on the trees, so that he can find a way out according to these marks when he gets lost one day in the future. Keeping track of past code changes is helpful for debugging future problems. If one day, your last modified program suddenly dies after running for a long time, then your first reaction at this time is what did I change, because it was good before the last modification. So how to detect this modification relative to the last time? Yes, the code control system SCS or version control system VCS (Concurrent Version Control, CVS is an evolution of VCS). Check in the previous version and compare it with the current test version. The comparison tool can be the diff tool that comes with SCS/VCS/CVS or other more powerful comparison tools, such as BeyondCompare and ExamDiff. By comparison, record all changed codes, and analyze all suspicious codes that may cause problems.

7 Determine the completeness of the test

How do you know how comprehensive your tests are? Coverage testing can answer this question. Coverage testing tools can tell you what code the CPU actually executes. A good coverage tool can usually tell you that about 20% to 40% of the code is correct, while the rest may contain bugs. The coverage tool has different test levels, and users can choose a certain level according to their needs. Even if you are confident that your unit tests are comprehensive and free of dead code, coverage tools can still point you to some potential problems, see the following code:

if (i >= 0 && (almostAlwaysZero == 0 || (last = i)))

If almostAlwaysZero is non-zero, then the last=i assignment is skipped, which may not be what you expect. This kind of problem can be easily found by the conditional testing function of the coverage tool.

In short, coverage testing is very helpful for improving code quality.

8 Improve code quality and save time

Studies have shown that more than 80% of software development time is spent on the following aspects:

• Debug your own code (unit tests)

• Debug yourself and other related code (inter-module tests)

• Test the whole system (system test)

What's worse is that you may need to spend 10-200 times as long to find a bug that may be easy to find in the beginning. A small bug can cost you dearly, and even if the bug doesn't have a big impact on the performance of the overall system, it is likely to affect the parts that you can see. Therefore, we must develop good coding and testing methods for higher code quality in order to shorten the code for debugging.

9 Spot it, analyze it, fix it

There is no panacea in this world. No matter how powerful the profile is, it is still powerless; no matter how good the memory monitor is, there are times when it cannot be found; no matter how easy the overlay tool is to use, there are places where it cannot be overwritten. Some deeply hidden problems may not be able to find the root cause even if all tools are used. At this time, what we can do is to find the regularity or anomaly through the external phenomena or some data output of these problems. Once any anomalies are found, it is important to understand them deeply and trace back to their root causes until they are resolved.

10 Leveraging a Beginner’s Mindset

Someone said: "Some things may have various situations in the minds of beginners, but they may be very single in the minds of experts." Sometimes, some simple problems are thought to be very complicated, and some simple systems are not designed to be very complicated, because of your "expert thinking". When you are stumped by a problem, turn off the computer, go for a walk, and talk about your problem with your friends or even your puppy, maybe they can give you unexpected inspiration.

Summary : Embedded debugging is also an art. Like any other art, if you want to be successful, you must have intelligence, experience and know how to use tools. As long as we can comprehend these ten secrets of Oracle well, I believe we will be able to succeed in embedded testing.

Finally:  The complete software testing video learning tutorial below has been sorted out and uploaded, and friends can get it for free if they need it【保证100%免费】

 These materials should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey. I hope it can help you too!