Chapter 1 System Verification
The reliability of ICT systems is a key issue
in the system design process. --The reliability of ICT systems is a key issue in the system design process.
1.0 Hard- and Software Verification
System verification technology is being applied to the design of ICT systems in a more reliable manner. This specification specifies what the system must and must not do and therefore forms the basis for any verification activity. Once the system fails to satisfy one of the specification properties, a defect is discovered. A system is considered "correct" as long as it satisfies all properties derived from its specifications. Therefore, correctness is always relative to the specification, rather than an absolute property of the system.
Software Verification : Peer reviewing and testing are the major software verification
techniques used in practice. – Software Verification Peer review and testing are the major software verification techniques used in practice. Peer review amounts to a software inspection by a team of software engineers who ideally had no involvement in the development of the software being reviewed. Empirical studies show that peer review provides an effective technique for detecting 31% to 93% of defects, with a median of approximately 60%. Due to its static nature, experience shows that subtle errors such as concurrency and algorithmic flaws are difficult to detect using peer review.
Software testing is an important part of any software engineering project. 30% to 50% of the total cost of a software project is spent on testing. As opposed to peer review, which statically analyzes code without executing it, testing is a dynamic technique that actually runs the software. Testing the software under consideration and providing input to its compiled code is called testing. Correctness is therefore determined by forcing the software to traverse a set of execution paths, sequences of code statements that represent the software's operation. Based on observations made during test execution, the actual output of the software is compared with the output recorded in the system specification. While test generation and test execution can be partially automated, comparisons are typically performed by humans. The main advantage of testing is that it can be applied to a variety of software, from application software (such as e-commerce software) to compilers and operating systems. Since exhaustive testing of all execution paths is not practical; in practice, only a small subset of these paths are processed. Therefore, the test can never be completed. That is, tests can only show the presence of a bug, not its absence. Another problem with testing is determining when to stop. In practice, it is difficult to indicate testing intensity to achieve a certain defect density, i.e. the proportion of defects per line of uncommented code, which is almost impossible.
About 50% of defects are introduced during programming, which is the stage where the actual coding occurs. While only 15% of bugs are detected during the initial design phase, most bugs are discovered during testing. Unit testing is designed to find defects in the individual software modules that make up the system, and a defect density of about 20 defects per 1000 lines of (uncommented) code is typical at the beginning of unit testing. At the beginning of system testing, there are about 6 defects per 1000 lines of code, where a set of modules that make up the real product are tested. When releasing a new software version, the generally accepted software defect density is about one defect per 1,000 lines of code1.
Bugs are usually concentrated in a few software modules - about half of the modules have no defects, and about 80% of the defects occur in a small proportion (about 20%) of the modules - and usually occur when connecting modules. Repair of bugs detected prior to testing can be accomplished quite economically. When a defect is only proven while the system is running, the cost of fixing increases significantly from about $1,000 in unit testing to a maximum of about $12,500 per bug fix. It is critical to pursue techniques that detect defects early in the software design process: the cost of fixing defects is greatly reduced, and they have less impact on the rest of the design.
Hardware Verification – Hardware verification to prevent hardware design errors is crucial. Hardware is expensive to manufacture; defects are difficult to fix after delivery to customers, and quality expectations are high.
·Hardware verification techniques. Emulation, simulation, and structural analysis are the
major techniques used in hardware verification. - Emulation, simulation, and structural analysis are the major techniques used in hardware verification.
Structural analysis includes several specific techniques, such as synthesis, timing analysis, and equivalence checking, which will not be described in detail here.
Simulation is a kind of test. A reconfigurable general-purpose hardware system (simulator) is configured to behave similarly to the circuit under consideration and then tested extensively. Like software testing, simulation amounts to providing a set of stimuli to a circuit and comparing the resulting output to the expected output specified in the chip's specifications. To fully test a circuit, all possible input combinations should be examined for every possible system state. This is impractical and requires a significant reduction in the number of tests, creating potential undiscovered bugs.
Simulation is the most popular hardware verification technique used at various design stages such as register transfer level, gate and transistor level. In addition to these error detection techniques, hardware testing is also required to detect manufacturing failures caused by layout defects during the manufacturing process
1.1 Model Checking
In software and hardware design of complex systems, verification takes more time and effort than construction. Look for techniques that reduce and simplify verification efforts while increasing their coverage. Formal methods have great potential to enable early integration of verification in the design process, provide more effective verification techniques, and reduce verification time.
Model checking is a verification technique that explores all possible system states in a brute force manner. Similar to a computer chess program that checks possible moves, a software tool that performs model checking, a model checker, checks all possible system scenarios in a systematic way. In this way, it can be proven that a given system model indeed satisfies a certain property. It is a real challenge to study the largest possible state space that can be processed by current methods, namely processors and memory. State-of-the-art model checkers can handle a state space of approximately 108 to 109 states via explicit state space enumeration. Using clever algorithms and customized data structures, larger state spaces (1020 to 10476 states) can be processed for specific problems. Even subtle errors that have not been discovered using simulation, testing, and emulation may be discovered through model checking
System models are usually automatically generated from a model description specified in some appropriate programming language (such as C or Java) or a hardware description language (such as Verilog or VHDL). Note that property specifications specify what the system should and should not do, whereas model descriptions describe the behavior of the system. The model checker examines all relevant system states to check whether they satisfy the required properties. If a state is encountered that violates the property under consideration, the model checker will provide a counterexample that indicates how the model reached the undesired state. Counterexamples describe the execution path from an initial system state to a state that violates the property being verified. With the help of the simulator, the user can replay the violation scenario and in this way obtain useful debugging information and adjust the model (or properties) accordingly (see Figure 1.4).
Model checking has been successfully applied to several ICT systems and their applications.
For example, deadlocks were detected in online airline reservation systems, modern e-commerce protocols were validated, and several studies on international IEEE standards for internal communication in home appliances led to significant improvements in system specifications. Five previously undiscovered bugs were discovered in the execution module of the Deep Space 1 spacecraft controller (see Figure 1.5), one of which revealed a major design flaw. During a flight experiment 96 million kilometers from Earth, a bug identical to one discovered by model checking escaped testing and caused a deadlock. In the Netherlands, model checks have found some serious design flaws in the control software for a storm surge barrier that protects Rotterdam's main port from flooding.
Example 1.1. Concurrency and Atomicity Most bugs, such as those exposed in the Deep Space One spacecraft, are related to classic concurrency bugs. Unforeseen intersections between processes can lead to unexpected events. This can be illustrated by analyzing the following concurrent program, in which the three processes Inc, Dec, an and Reset cooperate with each other. They operate on a shared integer variable x that has an arbitrary initial value that is accessible (i.e., read), and
Is the value of x always between (inclusive) 0 and 200? At first glance, this seems true. However, a more thorough examination showed that this was not the case. Suppose x equals 200. Procedure Dec tests the value of x, and passes the test because x exceeds 0. Then, control is taken over via a process reset. It tests the value of x, passes the test, and immediately resets x to zero. Control then returns to procedure Dec, which decrements x by 1, causing x to be negative (i.e. -1). Intuitively, we tend to interpret testing on x and assignment to x as being executed atomically, i.e. as a single step, when in fact (mostly) this is not the case.
1.2 Characteristics of Model Checking
Model checking is an automated technique that, given a finite-state model of a system and a formal property, systematically checks whether this property holds for (a given state in) that model.
Model checking is an automated technique that, given a finite-state model of a system and formal properties, systematically checks whether the properties hold true for (a given state in) the model.
1.2.1 The Model-Checking Process
• Modeling phase:
– model the system under consideration using the model description language of
the model checker at hand;
– as a first sanity check and quick assessment of the model perform some simu-
lations;
– formalize the property to be checked using the property specification language.
• Running phase: run the model checker to check the validity of the property in the
system model.
• Analysis phase:
– property satisfied? → check next property (if any);
– property violated? →
- analyze generated counterexample by simulation;
- refine the model, design, or property;
- repeat the entire procedure.
– out of memory? → try to reduce the model and try again.
Modeling: A system model describes the behavior of the system in an accurate and unambiguous way. They are mostly represented using finite state automata, consisting of a finite set of states and a set of transitions. State includes information about the current values of variables, previously executed statements (for example, the program counter), and so on.
Running the model checker: The model checker must first be initialized by appropriately setting the various options and directives that can be used to perform exhaustive verification. Subsequently, actual model checking is performed. This is basically a single algorithmic approach where the validity of the considered properties is checked in all states of the system model.
Analyzing the results: There are basically three possible outcomes: the specified properties are not valid in the given model, or the model is too large to fit within the physical limitations of computer memory.
Verification organization : The entire model checking process should be well organized, well structured, and well planned. Industrial applications of model checking demonstrate that the use of version and configuration management is of particular relevance. For example, during the verification process, different model descriptions are made for different parts of the system, various versions of the verification model are provided (e.g., due to abstraction), and a large number of verification parameters (e.g., model checking options) and results are provided (diagnostic tracking, statistics). In order to manage the actual model checking process and allow replication of the experiments performed, this information needs to be recorded and maintained very carefully
1.2.2 Strengths and Weaknesses
Advantages of model checking:
•It is a versatile verification method suitable for a wide range of applications such as embedded systems, software engineering and hardware design.
•It supports partial validation, i.e. properties can be checked individually, allowing to focus on basic properties first. A complete requirements specification is not required.
•The possibility of errors is less easily exposed; this contrasts with testing and simulations which are designed to track down the most likely defects.
•Provides diagnostic information when properties fail; this is useful for debugging
• It is a potentially "push-button" technology; using model checking requires neither a high degree of user interaction nor a high degree of expertise.
•Industry interest is growing rapidly; some hardware companies have started their in-house verification labs, job opportunities with the required skills in model checking are emerging frequently, and commercial model checking is already available.
• It can be easily integrated into existing development cycles; its learning curve is not very steep and empirical studies have shown that it may reduce development time.
•It has a sound mathematical foundation; it is based on graph algorithms, data structures and logic theory.
Disadvantages of model checking:
•It is mainly suitable for control-intensive applications and less suitable for data-intensive applications since data is usually distributed over an infinite domain.
• Its applicability depends on decidability problems; model checking is often not computationally efficient for infinite-state systems, or for reasoning about abstract data types (requiring undecidable or semi-decidable logic).
•It verifies the system model, not the actual system (product or prototype) itself; therefore, any results obtained are only as good as the system model. Supplementary techniques, such as testing, are needed to detect manufacturing faults (hardware) or coding errors (software).
•It only checks specified requirements, i.e. completeness is not guaranteed. There is no way to determine the validity of an unchecked property.
•It faces the state space explosion problem, whereby the number of states required to accurately model a system may easily exceed the amount of computer memory available. Although several very effective methods have been developed to solve this problem (see Chapters 7 and 8), models of real-world systems may still be too large to fit in memory.
• Its use requires some expertise in finding appropriate abstractions to obtain smaller system models and represent the properties in the logical form used.
•Not guaranteed to produce correct results: Like any tool, the model checker may contain software defects. 2.
•It does not allow inspection generalization: In general, inspection systems with an arbitrary number of components or parameterized systems cannot be handled. However, model checking can suggest results for arbitrary parameters verified using the proof assistant.
Model checking is an effective technique to expose potential design errors.