In embedded software development, utilizing full application traces offers developers unlimited possibilities for analyzing the behavior of their products. With full visibility into the application, they can trace every instruction to see if their application is functioning as expected, or if there are bugs or vulnerabilities. So, how can you get the most out of the existing RISC-V traces available?
What is tracking?
Compared to traditional debugging by setting breakpoints, printf, etc., tracing is more like watching your application without interruption. Basically, the developer can observe the entire work without interfering with the program. Trace includes complete instruction execution flow (no printf nor UART required), once trace data is captured, you can quickly trace back, isolate exceptions and hard faults.
This makes it easier to find those rare, execution-order-dependent bugs, because it provides a bunch of clues about how the program executes and in what order. This allows developers to see exactly how and why they always end up on a particular string of code. You can quickly find exceptions and hard faults, and thus find those rare, order-dependent errors. Without tracking, when your program crashes, it's very difficult to reproduce what actually happened.
However, tracing is not just for finding bugs. Tracing can do code profiling, code coverage, etc., and you can live-stream the behavior of your devices. Performance and coverage monitoring are powerful features, and when you have tracing capabilities, you can achieve these powerful features. If your bandwidth is wide enough, it's even possible to integrate a live trace stream into your debugger.
RISC-V Tracking
One of the main concerns of the RISC-V organization is to develop standardized specifications for RISC-V. Various working groups, open to all RISC-V members, are currently focused on developing these specifications. One such example is the Processor Trace Working Group, which approved the release of the Processor Trace Specification in February 2020. Another example is the Nexus Trace Group, which is making recommendations on how to use the trace defined by the Nexus IEEE-ISTO 5001™ standard for RISC-V cores.
This work will continue as all aspects of tracking standards must be considered. Among them, the output format of the trace control is included. The minimum goal is to be on par with existing standards of more mature architectures. If the RISC-V trace specification is done well, it will enable easy adoption of existing trace viewers, hardware trace probes, and trace analysis tools. There are already some implementations, but the RISC-V architecture should have a track in every device from the Internet of Things to servers. Even simple, standard tracking is better than no tracking at all.
Figure 1. Trace from a RISC-V device
see each instruction
Integrated support for tracing in software development tools enhances day-to-day code development/debugging. Tracing should be an inherent part of a designer's day-to-day environment, not an afterthought. So you can write code and run it to see how you got to the current point of execution. With the ability to quickly trace back and isolate exceptions and hard faults, you can iterate directly and achieve good code quality. You can also find bugs that depend on execution order and find power consumption measurements that can be correlated to your program flow. All these analyzes can also be performed in a multi-core environment due to its challenging and complex dependencies.
When looking for a vulnerability, flipping through captured traces is like looking for a needle in a haystack. In just a few seconds of execution time, hundreds of millions of instructions can be generated. It is therefore extremely important that the specification will provide sufficient triggers to be able to limit capture to specific areas. Advanced navigation and search capabilities are essential, and if your compiler/debugger tools provide this functionality, use trace triggers to limit trace data to what you need.
Why do I need to track?
Embedding a tracking unit in the device makes it possible for you to non-intrusively track the product while it is running. Even adding very low-level debug printouts can alter the timing of your application and obscure its true behavior.
There are a number of ways to capture and obtain trace data output from a device:
▫Sort by order
- Adequate PC sampling trace (good for statistical code profiling studies)
- Light instrumentation, real-time operating system (RTOS) monitoring, variable tracking, etc.
- With good probes, several megabytes/second speeds are possible
▫High-speed parallel interface (4 to 16-bit bilateral interface)
- capture everything (clock speed can be high)
- Tracking by messages left when control flow is transferred
- Guaranteed every instruction you execute
- Instruction execution process information is stored on the debugger probe in real time
▫RAM buffer
- Either small dedicated RAM or shared with system memory
- Even 4KB of trace RAM provides enough power to really work
▫High-speed arrangement
- Speed of 10Gbits/s or higher
- Primarily suitable for larger, complex systems
▫ Tracking via functional interface (USB3.0 provides ultra-high bandwidth!)
- Limited use cases - not an option for small IoT devices
Debug exceptions
By using traces in your day-to-day work, you can actually see how the application is running. You can examine program flow down to a specific state, such as an application crash, and use the trace data to locate the source of the problem.
Exceptions or unhandled faults can be caused by pointer problems, illegal instructions, or data aborts. Normally, your stack (and function call information) will be destroyed when this happens, but with trace, you can get the complete application history. Trace data is also useful for locating programming errors that appear sporadically and sporadically. This can help uncover "million dollar" vulnerabilities.
Integration of hardware and software tools
The best way to get the highest quality code possible is to integrate trace analysis capabilities into your daily development environment. If you can verify program timing and correctness every time a deployment changes, the risk of pushing complex bugs to later system verification or even customers is greatly reduced. The best solution is to have hardware tools like trace debug probes that work seamlessly with software tools in your IDE. For example, IAR offers the trace probe I-jet Trace, which is fully integrated with the complete C/C++ compiler and debugger toolchain IAR Embedded Workbench.
I-jet Trace unlocks some advanced features in the IAR Embedded Workbench integrated development environment. In Figure 2, you can see several examples of high-level view functionality, with the timeline above and the code stack below. This shows that the program is running, the program is making complex function calls, and the interrupt is also triggered synchronously. The blue part shows the data trace, so you can use the instrumentation and see the RTOS switching and so on. The detailed graph below shows the startup code in action - there are complex calls, and some longer and shorter functions, and there's even a tooltip to optimize the code for one of the calls.
Figure 2. Example timeline with call stack and interrupt and variable logging
code quality
But tracking isn't just about finding vulnerabilities. It also helps you monitor your application - is it behaving as expected? Integrated performance monitoring is one of the main benefits of using traces, as it can help you understand where your application execution time is being spent, whether it is affected by interrupt storms, whether it is sometimes not responsive enough, etc.
Code coverage is also a feature that can be implemented with tracing (Figure 3 shows how it looks in IAR Embedded Workbench). This can be used to prove that the code was run at least once, to find code that was not run, to show test defects, etc. Functional safety certification strongly recommends code coverage as a means of improving quality. Static code analysis tools are a great complement to trace analyzers. This ensures that the code complies with industry-specific standards and best programming practices. In addition to coverage, the tool can also collect the number of executions of each individual instruction. For example, some instruction blocks are executed 4 times, some are executed 12 times, and the code that is not covered is executed 0 times.
Figure 3. Tracing can also review code coverage
Summarize
Tracing provides a variety of beneficial data sources, such as collecting data or assisting developers in filtering information into actual knowledge and insights. Visualization and analysis with different trace viewers creates a realistic picture of how the product operates and its dynamic behavior, providing developers with the necessary information to track down complex vulnerabilities that are difficult, if not impossible, to catch by other means. By incorporating tracing into daily development, developers can speed up the software development process and improve software quality.