RSI Logo

Model-Based Testing and Validation with Reactis®

 
Products |Services |Support |Company |News |FAQ
 Reactis User's Guide   Contents  |  Index
 Chapters:  1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20

Chapter 6  Reactis Coverage Criteria

Reactis uses a number of different coverage criteria on Simulink / Stateflow models to measure how thoroughly a test or set of tests exercises a model. In general, coverage criteria record how many of a given class of syntactic constructs, or coverage targets, that appear in a model have been executed at least once. Some of the criteria supported by Reactis involve only Simulink, some are specific to Stateflow, and the remaining are generic in the sense that they include targets within both the Simulink and the Stateflow portions of a model. When using Reactis for C Plugin, coverage targets are also tracked in the C code portions of models (S-Functions and Stateflow custom C code). The criteria discussed in this chapter may be visualized using Simulator and are central to test generation and model validation using Tester and Validator.

6.1  Simulink-Specific Criteria

The Simulink-specific criteria include the following: conditional subsystem coverage, branch coverage, and lookup table coverage.

6.1.1  Conditional Subsystem Coverage

A (conditional) subsystem is deemed covered if has been executed at least once in some simulation step. In general, every subsystem within a Simulink diagram executes during every simulation step. However, conditional subsystems may be disabled during a simulation step and hence not execute.

6.1.2  Branch Coverage

A number of different Simulink blocks contain targets included in the branch coverage measure. Each block in this group has the characteristic that the set of all possible outcomes of evaluating the block may be partitioned into well-defined sets of mutually exclusive outcomes. For example, the possible outcomes of evaluating a Logical Operator block are true or false. Therefore, for each Logical Operator block, we consider the true outcome and the false outcome to be the two branches associated with the block. Each branch becomes a coverage target in the branch-coverage criterion.

The following discusses branch coverage in more detail. Reactis Tester aims to exercise all such branches in the test suites it generates. Reactis Simulator includes several ways to track the branches that have been exercised. In particular, in model diagrams, every block that includes branches is drawn in a way to show each branch. Simulator may be configured to render uncovered branches in red. We now list the blocks included in the branch-coverage criterion; for each block, we describe its associated branches and how coverage information for the branches is drawn in Simulator.

Logical Operator.
The branches are:
  1. the block evaluates to true
  2. the block evaluates to false
Relational Operator.
The branches are:
  1. the block evaluates to true
  2. the block evaluates to false
branchCvg1.1.png
Figure 6.1: Branches in Relational and Logical Operator blocks.
Switch.
Let C be the condition for passing the first input of a Switch block; i.e. C is u2 ≥ threshold, u2 > threshold, or u2 ~= 0. Then the branches are:
  1. C evaluates to true.
  2. C evaluates to false.
Multiport Switch.
If a multiport switch has n data inputs, then it is considered to have n branches, one for each data input. Branch i is covered when the control input has value i during evaluation of the block.
branchCvg2.1.png
Figure 6.2: Branches in Switch and Multiport Switch blocks. Each branch target in a Switch block or a Multiport Switch block is drawn as a line from an input of the block to the output of the block. The line from an input represents the branch where that input is taken as output for the block.
Dead Zone.
Let X be the input to, L the lower limit of, and U the upper limit of a Dead Zone block. Then the branches are:
  1. X <= L
  2. L < X < U
  3. X >= U
Saturation.
Let X be the input to, L the lower limit of, and U the upper limit of a Saturation block. Then the branches are:
  1. X < L
  2. L <= X <= U
  3. X > U
branchCvg3.1.png
Figure 6.3: Branches in Dead Zone and Saturation blocks. Each line segment represents a branch.
Discrete-Time Integrator.
Let V be the integrated value, L the lower limit of, and U the upper limit of a Discrete-Time Integrator block. Then the branches are:
  1. V < L
  2. L <= V <= U
  3. V > U
branchCvg10.1.png
Figure 6.4: Branches in Discrete-Time Integrator blocks. Each line segment represents a branch.
If.
The Block Parameters dialog for the If block includes a “Show else condition” check box that determines whether or not the block includes an output port for the else case of the block. The definition of the branches for this block differ slightly depending on the value of this setting.
  • If an If block has n outputs and the “Show else condition” check box is selected, then the block is considered to have n branches, one for each output. Branch i is covered when the system connected to output i executes.
  • If an If block has n outputs and the “Show else condition” check box is not selected, then the block is considered to have n + 1 branches, one for each output and one for the implicit else case. For branches 1 to n, branch i is covered when the system connected to output i executes. Branch n + 1 is covered when the If block executes, but no output fires (that is when the implicit else case is taken).
branchCvg6.1.png
Figure 6.5: Branches in If blocks.
Switch Case.
The Block Parameters dialog for the Switch Case block includes a “Show default case” check box that determines whether or not the block includes an output port for the default case. Reactis tracks a branch for the default chase whether or not this check box is selected. Additionally the parameters of this block define a sequence of integer sets 1 S1, S2, ..., Sn. Each set is associated with an output of the block. Reactis tracks a branch target for each element of each set Si. A branch value n in set Si is covered if the control input to the block is n and the outport associated with Si fires.
branchCvg9.1.png
Figure 6.6: Branches in Switch Case blocks.
Enable.
Let X be the input to an Enable port of the subsystem S in which the Enable block resides. Then the branches are:
  1. X > 0 - subsystem S executes
  2. X ≤ 0 - subsystem S does not execute
branchCvg7.1.png
Figure 6.7: Branches in Enable block.
MinMax.
If a MinMax block has n inputs, then it is considered to have n branches, one for each input. Branch i is covered when input i is selected as the minimum or maximum value of inputs to the block.
branchCvg4.1.png
Figure 6.8: Branches in a MinMax block.
Relay.
The branches are
  1. The relay is on.
  2. The relay is off.
branchCvg5.1.png
Figure 6.9: Branches in a Relay block.
Abs and Sign blocks.
Let X be the input to an Abs block. Then the branches are:
  1. X ≥ 0
  2. X < 0

Let X be the input to a Sign block. Then the branches are:

  1. X > 0
  2. X = 0
  3. X < 0
branchCvg8.1.png
Figure 6.10: Branches in Abs and Sign blocks.

6.1.3  Lookup Table Coverage

Reactis supports coverage tracking for the Lookup Table and Lookup Table (2-D) blocks. Intuitively a coverage target will be allocated for each interval specified by the input settings of a table. To view the coverage information for a table in Simulator, after selecting Coverage -> Show Details, right-click on the table and select View Coverage Details.

1-D Tables

If a Lookup Table block (one-dimensional) has inport C and “Vector of input values” [ c1, c2, ... , cn ] then the block will have the following targets:

C < c1c1 ≤ C < c2c2 ≤ C < c3...cn−1 ≤ C < cnC ≥ cn
lookupTableCvg.1.png
Figure 6.11: Lookup table targets for 1-D tables. Each column has a header of the form [ci,ci+1) to indicate the target interval ci ≤ C < ci+1. The table shown may be displayed in Simulator (after selecting Coverage -> Show Details) by right clicking on the lookup table block and selecting View Coverage Details.

2-D Tables

If a Lookup Table (2-D) block has inputs R and C, “Row index input values” [ r1, r2, ... , rm ], and “Column index input values” [ c1, c2, ... , cn ] then the block will have the following targets:

R < r1 and C < c1R < r1 and c1 ≤ C < c2R < r1 and c2 ≤ C < c3...R < r1 and cn−1 ≤ C < cnR < r1 and C ≥ cn
r1 ≤ R < r2 and C < c1r1 ≤ R < r2 and c1 ≤ C < c2r1 ≤ R < r2 and c2 ≤ C < c3...r1 ≤ R < r2 and cn−1 ≤ C < cnr1 ≤ R < r2 and C ≥ cn
r2 ≤ R < r3 and C < c1r2 ≤ R < r3 and c1 ≤ C < c2r2 ≤ R < r3 and c2 ≤ C < c3...r2 ≤ R < r3 and cn−1 ≤ C < cnr2 ≤ R < r3 and C ≥ cn
⋮
rm−1 ≤ R < rm and C < c1rm−1 ≤ R < rm and c1 ≤ C < c2rm−1 ≤ R < rm and c2 ≤ C < c3...rm−1 ≤ R < rm and cn−1 ≤ C < cnrm−1 ≤ R < rm and C ≥ cn
R ≥ rm and C < c1R ≥ rm and c1 ≤ C < c2R ≥ rm and c2 ≤ C < c3...R ≥ rm and cn−1 ≤ C < cnR ≥ rm and C ≥ cn
lookupTableCvg2.1.png
Figure 6.12: Lookup table targets for 2-D tables. Each column has a header of the form [ci,ci+1) to indicate the target interval ci ≤ C < ci+1. Each row has a header of the form [rj,rj+1) to indicate the target interval rj ≤ R < rj+1. The table shown may be displayed in Simulator (after selecting Coverage -> Show Details) by right clicking on the lookup table block and selecting View Coverage Details.

6.2  Stateflow-Specific Criteria

The Stateflow criteria are defined with respect to the graphical syntax of Stateflow. Among other things, Stateflow diagrams contain states, transition segments, and junctions. Each transition segment in turn may have a label that includes all or some of the following: an event, a condition, a condition action, and a transition action. A segment’s condition action is executed whenever the segment is deemed ready to fire. A transition consists of a sequence of segments leading from one state to another. The transition will fire when each segment in the sequence is ready to fire. A segment’s transition action is executed only when it is included in such a firing transition. The Stateflow criteria may now be defined as follows.

State coverage.
The targets in this criterion are states; a state is covered if it has been entered at least once.
Condition action coverage.
The targets in this criterion are transition segments; a segment is deemed to be covered if its condition action has been executed at least once, or, if the segment has no condition action, if its condition has evaluated to “true” at least once. Note that a segment with an empty condition is assumed to have a condition that always evaluates to “true”.
Transition action coverage.
The targets in this criterion are transition segments that have transition actions; such a segment is deemed to be covered if its transition action has been executed at least once. Note that if a segment has no transition action, it is ignored by this criterion.
sfCvg.1.png
Figure 6.13: Highlighting of the Stateflow coverage criteria in Simulator. Note that in addition to condition-action and transition-action targets a transition segment may additionally have MC/DC targets.

6.3  Generic Criteria

Generic coverage criteria define targets that may appear in either the Simulink, the Stateflow, or the C code 2 portions of a model. Condition, Decision, MC/DC, and Boundary coverage are based on well-known criteria of the same names developed for measuring coverage of source code.

6.3.1  Condition, Decision and MC/DC Criteria

Reactis includes facilities for generating tests to meet the Condition, Decision and Modified Condition / Decision Coverage (MC/DC) requirements. These criteria involve boolean-valued structures (conditions, decisions) within a model; understanding these criteria requires that the notions of conditions and decisions be defined precisely. As the notions were first developed in the context of traditional programming languages (e.g. C, Ada), the next paragraphs first review their traditional definitions. The adaptations of these notions to Simulink and Stateflow are then discussed.

In software testing, a decision is a boolean-valued expression used to determine which execution path to follow. Conditions represent the atomic (i.e. smallest) boolean expressions within a decision. Decisions typically are constructed using several conditions. As an example, consider the following statement from a C program. 

    if ((x > 0) && (y > 0))
       z = 1;
    else
       z = 2;

Here ((x > 0) && (y > 0)) is a decision, since the value of the expression determines whether z is assigned the value 1 or 2. Both (x > 0) and (y > 0) are conditions, since they are boolean-valued expressions that cannot be broken into smaller boolean expressions.

Traditional Decision coverage may now be defined as follows. Each decision in a program gives rise to two targets: the evaluation of the decision to true, and to false; a program is fully covered by a test suite when each target has been covered, i.e. each decision has evaluated to both true and false. Condition coverage is defined very similarly; the difference is that each condition evaluates to both true and false, rather than each decision.

MC/DC is somewhat more complex to define. It was introduced by John J. Chilenski of Boeing in the early 90s; the definitive research paper was published by Chilenski and Steve Miller, of Rockwell-Collins, in 1994. MC/DC is the level of testing mandated by the Federal Aviation Administration (FAA) in its DO-178/B guidelines for the “most safety-critical” components of aviation software. The MC/DC targets in a program are the conditions; a condition C in decision D is covered by a test suite if there are two test steps X and Y (not necessarily consecutive) in the suite such that:

  • C evaluates to a different truth value (true or false) in step X than in step Y; and
  • each condition other than C in D evaluates to the same truth value in both step X and step Y; and
  • D evaluates to a different truth value in step X than in step Y.

In other words, each condition must be shown to independently affect the outcome of its enclosing decision.

Short-circuiting of boolean expressions can affect the difficulty of achieving full MC/DC coverage of a program. A short-circuited operator avoids evaluating all conditions of a decision if the outcome is determined after evaluating only a subset of the conditions. For example, a short-circuited “and” operator would not evaluate its second argument if the first evaluates to false. (Virtually all programming languages use short-circuiting for reasons of efficiency.) Without going into the technical details, it can be said that achieving full MC/DC coverage is easier if short-circuited boolean operators are used. Section 4.7.1 explains how Reactis may be instructed to treat Simulink / Stateflow boolean operators as short-circuited.

In order to adapt the notions of Decision, Condition and MC/DC coverage to Simulink / Stateflow, it suffices to define what the conditions and decisions in a model are. These are as follows.

  • For Logical Operator blocks, Reactis supports two alternative modes for tracking Decision, Condition, and MC/DC coverage. The mode is set from the Reactis Settings Dialog (see Section 4.7). The two modes are:
    Single-Block.
    The output of each Logical Operator block is a decision and its inputs constitute the conditions within the decision.
    Multi-Block.
    Multiple Logical Operator blocks that are directly connected are grouped into a single decision. The inputs to blocks in the group that are not connected to the outputs of other blocks in the group constitute the conditions of the decision.

    When Multi-Block MC/DC is enabled, the following rules are used to combine the Logical Operator blocks of a model into groups, each of which constitutes a single decision:

    • Groups may only contain Logical Operator blocks.
    • Groups may not cross subsystem boundaries.
    • Each group has exactly one exiting signal line. The output port of the block at which the exiting signal line originates will be marked as the decision. A red dot on the output port indicates that the decision has not evaluated to both true and false. The coverage details table for the decision may be invoked by right-clicking on the output port and selecting View Coverage Details. Hovering over the block causes blue lines to be drawn from each condition in the decision to the output port.
    • Each group may have multiple incoming signal lines. The input ports of the blocks where the entering signal lines end will be marked as the conditions of the decision. A red dot on one of the inports implies that the port has an uncovered condition or MC/DC target.
    • The set of Logical Operator blocks L within a subsystem are partitioned into a set of groups as follows. For each l ∈ L, let G(l) denote the group containing l.
      1. For each li in L, let G(li) = { li }
      2. For each li ∈ L: If all branches of the output of li feed into a group gj then merge G(li) and gj otherwise G(li) remains unchanged.
  • The “if expression” and “elseif expressions” within an If block in Simulink are decisions. In this case the atomic boolean-valued subexpressions are the conditions.
  • The event / guard combinations on Stateflow transitions (such decisions evaluate to true if the event is present and the guard is true, and to false otherwise) are decisions. In this case, the conditions within the decision associated with a transition segment are:
    • An Event on the transition segment. (the event is “true” if it is present and “false” otherwise)
    • The atomic boolean-valued expressions in the guard of the segment.
  • When using Reactis for C Plugin additional decisions are tracked as described in Chapter 16.

Coverage information for Decision, Condition, and MC/DC coverage is rendered as shown in Figure 6.14. Right-clicking on an outport of a Logical Operator block and selecting View Coverage Details gives additional information.

decCondCvg.1.png
Figure 6.14: Visualization of MC/DC related coverage information.

6.3.2  Boundary Value Coverage

Boundary Value Coverage tracks whether a data item assumes values of interest for a particular block. In the case of top-level model inports and configuration variables, these are the boundary values of a block’s domain of possible values. In the case of a Relational Operator block, the values of interest occur when the two inputs to the block are equal or very close.

Boundary Values for Inports and Configuration Variables

The boundary values tracked for an inport or configuration variable are determined by its associated type 3 as shown in Table 6.1. If an inport or configuration variable has a type not shown in the table, then it has no boundary value targets.

boundval_cvg.1.png
Figure 6.15: Shows uncovered boundary values for inports. Reactis will attempt to cover the limits of the specified port, depending on its type.
Table 6.1: Boundary values associated with each type.
RSI TypeBoundary Values
t[i, j], where t is double, single, sfix*, or ufix*if i < 0 and j > 0 then i, 0.0, j; otherwise i, j
t[i, j], where t is and integer typeif i < 0 and j > 0 then i, i+1, 0, j−1, j; otherwise i, i+1, j−1, j
t{ e1, … en }e1, … en
t [i:j:k]if there exists a positive integer l such that i + l × j = 0.0 then i, i+j, 0.0, k−j, k; otherwise i, i+j, k−j,k
t delta [i,j]boundary values of t
booleantrue, false
int8-128, -127, 0, 126, 127
int16-32768, -32767, 0, 32766, 32767
int32-2147483648, -2147483647, 0 2147483646, 2147483647
uint80, 1, 254, 255
uint160, 1, 65534, 65535
uint320, 1, 4294967294, 4294967295
double0.0
single0.0

Boundary Values for Relational Operators

Reactis also tracks boundary value coverage for the Relational Operator block. For a relational operator block with inputs a and b, the boundary value targets are:

  • a = b
  • a = b + є
  • a = b − є

where є is the smallest increment representable in the type of a and b. For floating-point values, the user can specify the smallest є, by typing “Next” (i.e. next increment) or specify a relative tolerance є by typing the value in the given dialog box. Note that these coverage criteria have the potential to introduce a large number of very hard to cover targets. Since some users might prefer to not track such targets, this functionality is disabled by default and can be enabled from the Reactis Settings dialog.

6.4  Validator-Related Targets

See Chapter 9 for a description of the two Validator-related targets: assertions and user-defined targets.

1
Some sets may contain only a single value.
2
C code coverage tracking becomes enabled when using the Reactis for C Plugin; otherwise coverage is not tracked in the C code portions of a model.
3
Recall that types are associated with top-level inports using the Port Types tab of the Reactis Info File Editor (described in Chapter 5).