| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Chapter 6 Reactis Coverage CriteriaReactis uses a number of different coverage criteria on Simulink / Stateflow models to measure how thoroughly a test or set of tests exercises a model. In general, coverage criteria record how many of a given class of syntactic constructs, or coverage targets, that appear in a model have been executed at least once. Some of the criteria supported by Reactis involve only Simulink, some are specific to Stateflow, and the remaining are generic in the sense that they include targets within both the Simulink and the Stateflow portions of a model. The criteria discussed in this chapter may be visualized using Simulator and are central to test generation and model validation using Tester and Validator. 6.1 Simulink-Specific CriteriaThe Simulink-specific criteria include the following conditional subsystem coverage, branch coverage, and lookup table coverage. 6.1.1 Conditional Subsystem CoverageA (conditional) subsystem is deemed covered if has been executed at least once in some simulation step. In general, every subsystem within a Simulink diagram executes during every simulation step. However, conditional subsystems may be disabled during a simulation step and hence not execute. 6.1.2 Branch CoverageA number of different Simulink blocks are tracked by the branch coverage measure. Each block in this group has the characteristic that the set of all possible outcomes of evaluating the block may be partitioned into well-defined sets of mutually exclusive outcomes. For example, the possible outcomes of evaluating a Logical Operator block are true or false. Therefore, for each Logical Operator block, we consider the true outcome and the false outcome to be the two branches associated with the block. Each branch becomes a coverage target in the branch-coverage criterion. The following discusses branch coverage in more detail. Reactis Tester aims to exercise all such branches in the test suites it generates. Reactis Simulator includes several ways to track the branches that have been exercised. In particular, in model diagrams, every block that includes branches is drawn in a way to show each branch. Simulator may be configured to render uncovered branches in red. We now list the blocks included in the branch-coverage criterion; for each block, we describe its associated branches and how coverage information for the branches is drawn in Simulator.
6.1.3 Lookup Table CoverageReactis supports coverage tracking for the Lookup Table and Lookup Table (2-D) blocks. Intuitively a coverage target will be allocated for each interval specified by the input settings of a table. To view the coverage information for a table in Simulator, after selecting Coverage -> Show Details, right-click on the table and select View Coverage Details. 1-D TablesIf a Lookup Table block (one-dimensional) has inport C and “Vector of input values” [ c1, c2, ... , cn ] then the block will have the following targets:
2-D TablesIf a Lookup Table (2-D) block has inputs R and C,
“Row index input values” [ r1, r2, ... , rm ], and
“Column index input values” [ c1, c2, ... , cn ] then the block
will have the following targets:
6.2 Stateflow-Specific CriteriaThe Stateflow criteria are defined with respect to the graphical syntax of Stateflow. Among other things, Stateflow diagrams contain states, transition segments, and junctions. Each transition segment in turn may have a label that includes all or some of the following: an event, a condition, a condition action, and a transition action. A segment’s condition action is executed whenever the segment is deemed ready to fire. A transition consists of a sequence of segments leading from one state to another. The transition will fire when each segment in the sequence is ready to fire. A segment’s transition action is executed only when it is included in such a firing transition. The Stateflow criteria may now be defined as follows.
6.3 Generic CriteriaGeneric coverage criteria define targets that may appear in either the Simulink or the Stateflow portions of a model. Condition, Decision, MC/DC, and Boundary coverage are based well-known criteria of the same names developed for measuring coverage of source code. Assertion and User-Defined Target targets may be added to a model using Reactis Validator are discussed more there. 6.3.1 Condition, Decision and MC/DC CriteriaReactis includes facilities for generating tests to meet the Condition, Decision and Modified Condition / Decision Coverage (MC/DC) requirements. These criteria involve boolean-valued structures (conditions, decisions) within a model; understanding these criteria requires that the notions of conditions and decisions be defined precisely. As the notions were first developed in the context of software testing, the next paragraphs first review their traditional definitions. The adaptations of these notions to Simulink and Stateflow are then discussed. In software testing, a decision is a boolean-valued expression used to determine which execution path to follow. Conditions represent the atomic (i.e. smallest) boolean expressions within a program. Decisions typically are constructed using several conditions. As an example, consider the following statement from a C program. if ((x > 0) && (y > 0))
z = 1;
else
z = 2;
Here Traditional Decision coverage may now be defined as follows. Each decision in a program gives rise to two targets: the evaluation of the decision to true, and to false; a program is fully covered by a test suite when each target has been covered, i.e. each decision has evaluated to both true and false. Condition coverage is defined very similarly; the difference is that each condition is assigned two targets, rather than each decision. MC/DC is somewhat more complex to define. It was introduced by John J. Chilenski of Boeing in the early 90s; the definitive research paper was published by Chilenski and Steve Miller, of Rockwell-Collins, in 1994. MC/DC is the level of testing mandated by the Federal Aviation Administration (FAA) in its DO-178/B guidelines for the “most safety-critical” components of aviation software. The MC/DC targets in a program are the conditions; a condition C in decision D is covered by a test suite if there are two test steps X and Y (not necessarily consecutive) in the suite such that:
In other words, each condition must be shown to independently affect the outcome of its enclosing decision. Short-circuiting of boolean expressions can affect the difficulty of achieving full MC/DC coverage of a program. A short-circuited operator avoids evaluating all conditions of a decision if the outcome is determined after evaluating only a subset of the conditions. For example, a short-circuited “and” operator would not evaluate its second argument if the first evaluates to false. (Virtually all programming languages use short-circuiting for reasons of efficiency.) Without going into the technical details, it can be said that achieving full MC/DC coverage is easier if short-circuited boolean operators are used. Section 4.6.1 explains how Reactis may be instructed to treat Simulink / Stateflow boolean operators as short-circuited. In order to adapt the notions of Decision, Condition and MC/DC coverage to Simulink / Stateflow, it suffices to define what the conditions and decisions in a model are. These are as follows.
6.3.2 Boundary Value CoverageBoundary Value Coverage tracks whether a data item assumes values of interest for a particular block. In the case of top-level model inports and configuration variables, these are the boundary values of a block’s domain of possible values. In the case of a Relational Operator block, the values of interest occur when the two inputs to the block are equal or very close. Boundary Values for Inports and Configuration VariablesThe boundary values tracked for an inport or configuration variable are determined by its associated type 2 as shown in Table 6.1. If an inport or configuration variable has a type not shown in the table, then it has no boundary value targets.
Boundary Values for Relational OperatorsReactis also tracks boundary value coverage for the Relational Operator block. For a relational operator block with inputs a and b, the boundary value targets are:
where є is the smallest increment representable in the type of a and b. Note that this coverage criterion has the potential to introduce a large number of very hard to cover targets. Since some users might prefer to not track such targets, this functionality is disabled by default and can be enabled from the Reactis Settings dialog. 6.3.3 Validator-Related TargetsSee Chapter 9 for a description of the two Validator-related targets: assertions and user-defined targets. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||