Reactis Validator searches for defects and inconsistencies
in models. The tool enables users to formulate a property that model
behavior should
have as an
assertion, attach the assertion to a model, and perform an automated
search for a violation of the
assertion. If Validator finds an assertion violation, it returns a
test that leads to the problem. This test may then be executed in
Reactis Simulator to gain an understanding of the sequence of
events that leads to the problem.
Validator also allows users to specify specific test scenarios they want
exercised in the form of user-defined targets. These targets may be seen as
user-defined extensions to the built-in coverage criteria supported by
Reactis: in generating test data, Validator (and Tester) will
attempt to generate test runs that satisfy the indicated scenarios. Like
assertions, user-defined targets are attached to models.
Validator is particularly useful in requirements validation. Given a list
of requirements on model behavior, users can formulate assertions (to check
whether a requirement is being satisfied) and user-defined targets (to define
test scenarios intended to “stress” the requirement).
Engineers use Validator as follows. First a model is instrumented
with assertions to be checked and/or user-defined coverage targets.
In the following discussion we shall refer to such assertions and
coverage targets as Validator objectives.
The tool is then invoked on the instrumented model to
search for assertion violations and paths leading to the specified
coverage targets. The output of a Validator run is a test suite
that includes tests leading to objectives found during the analysis.
Validator objectives may be added to any Simulink system or
Stateflow diagram in a model. When adding them to a Simulink system,
three mechanisms for formulating objectives are supported:
-
Expression objectives
- are C-like boolean expressions.
- Timer objectives
- are directives that tag a model data item
as a timer or counter.
- Diagram objectives
- are references to subsystems in standard
Simulink/Stateflow libraries.
An expression objective is a C-like boolean expression whose free
variables are the names of data items from the context in which the
objective is attached. Expression objectives are easily attached and
modified from the Reactis GUI. For more information on valid
expressions see Section 9.2.1.
A timer objective tags a model variable as a timer or counter. In the
case of a timer user-target, the target is considered covered after
the data item changes from its start value to its end value by a
specified increment. In the case of a timer assertion, the assertion
is considered violated if the data item ever changes from its start
value to the end value by a specified increment.
Diagram objectives give users the full power of Simulink / Stateflow to formulate
assertions and targets. The objectives may use any Simulink blocks
supported by Reactis, even and especially
Stateflow diagrams.
Diagram objectives are attached to a model using the Reactis GUI to
specify a Simulink system from a library and “wire” it into the
model. The diagrams are created using Simulink and Stateflow in
the same way standard models are built. After adding a diagram
objective to a model, the diagram is included in the model’s
hierarchy tree, as are other library links in a model.
9.1 The Meaning of Validator Objectives
We now explain in more detail how Validator interprets objectives.
Coverage of these objectives may be tracked in the standard manner,
namely through highlighting in the main Reactis panel or in the
Coverage-Report Browser. Also, in Simulator, hovering over a covered
objective shows the test and step number when it was first covered.
9.1.1 Assertions
Assertions specify properties that should always be true for a model.
For a given assertion, Validator searches a model for a simulation
run that leads to a configuration in which the assertion does not
hold. An expression assertion fails to hold if the given expression
evaluates to false1. A timer
assertion fails to hold if the data item it tracks ever changes
from its start value to the end value by a specified increment. A
diagram assertion fails to hold if any output port assumes the boolean
value “false” or a numeric value zero. Note that if a diagram has
more than one output port, the individual port names are listed in the
target list and highlighted independently according to whether they
are violated or not.
An assertion is considered “covered” when a violation is
found. So in contrast to targets, where being covered is considered
good, covering an assertion is bad. We therefore highlight covered assertions in red and yellow, in contrast to targets where
uncovered targets are highlighted in red.
9.1.2 User-Defined Coverage Targets
User-defined coverage targets extend the Reactis built-in targets
described in Chapter 6 (branch coverage, state
coverage, condition-action coverage, MC/DC, etc). If a user-defined
target is specified as an expression, Reactis will treat it as
covered when the expression evaluates to a non-zero numeric value.
If a user-defined target is specified as a timer, the target is
considered covered after the data item tracked by the timer changes
from its start value to its end value by its specified increment
If a user-defined target is specified as a diagram, Reactis will
treat it as covered when all output ports of the associated
subsystem have assumed either a boolean “true” value or any non-zero
numeric value. Note that if a diagram has more than one output port,
the individual port names are listed in the target list and
highlighted independently according to whether they are covered or
not.
9.2 Adding, Editing, and Removing Objectives
Adding, editing, and removing of objectives is only possible when
Simulator is disabled. This is necessary because objectives
are linked into a model when Simulator is invoked. However,
operations with no semantic effect, such as moving or resizing
objectives, are possible even when Simulator is active.
A Validator objective may be added to a model by first selecting
in the hierarchy panel the Simulink subsystem or Stateflow
state in which the objective should be inserted and then:
-
right-clicking in an empty space in the subsystem in the
main panel of the Reactis window and selecting
“Add Assertion” or
“Add User-Defined Target”;
or
- by selecting the Validate -> Add Assertion or
Validate -> Add User-Defined Target menu items from
the main menu bar.
Usage of the resulting dialogs is explained in Section 9.2.1
(expression objectives), Section 9.2.2 (timer objectives),
and Section 9.2.3 (diagram objectives).
In Stateflow charts, only expression objectives are
supported. Using a group of radio buttons in the
wiring dialog, one can choose for the objective to be evaluated at state entry
or exit,
or every time the chart is triggered while the state is active. These
correspond to the usual entry, exit, and during actions of
Stateflow. Evaluation of the objective occurs at the end of the
corresponding state actions (entry, during, exit).
To edit properties of an objective, either right-click on the
objective and select “Edit”, or click on the objective to
select it and then choose the Validate -> Edit Assertion menu
item. Note that all the controls in the properties dialog are
disabled if Simulator is currently active. To remove an objective,
either:
-
right-click on the objective and select “Remove”, or
- click on the objective to select it and then choose the
Validate -> Remove Objective menu item, or
- click on the objective to select it and then press the delete key.
Standard cut, copy, and paste operations are available for objectives
both from the top-level Edit menu and by right-clicking on
an objective. Objectives may be moved by dragging them with the mouse.
9.2.1 The Expression Objective Dialog
The numbers below refer to the labels in Figure 9.1.
| Figure 9.1: The dialog for editing expression objectives. |
-
Objective name. The name must be unique with respect to the subsystem
where the objective resides.
- Expression. Reactis evaluates this expression to
determine whether the objective has been covered. The result
must be a scalar value of numeric type.
Free variables in the expression are the names of data items
(Simulink blocks or Stateflow variables) visible at the
place in the model where the objective is attached. Block names
that include whitespace or other special characters must be
enclosed in quotes. If a block name includes a quotation mark,
the quotation mark must be prefixed by a backslash. The notation
for accessing outport bar of subsystem foo is
“::bar@”. To simplify expression construction
users may insert variables by double clicking on an item
in the “Visible Variables” list (window item 3).
Valid operators and functions to use in the expression are:
+, -, *, / | Addition, Subtraction, Multiplication, Division |
<, > | Less than, Greater than |
<=, => | Less than or equal to, Greater than or equal to |
== | Equal |
!=, ~= | Not equal |
! | Logical Not |
|, || | Logical Or |
&, && | Logical And |
[] | Vector element access |
abs(), fabs() | Absolute value |
^, pow(,), power(,) | x to the power of y |
exp() | Exponent |
ln(), log(), log10() | Logarithm |
sqrt() | Square root |
rem() | Division remainder |
inf | Infinity value |
floor(), ceil() | Rounding |
cos(), sin(), tan() | trigonometric functions |
cosh(), sinh(), tanh() | hyperbolic trigonometric functions |
acos(), asin(), atan() | inverse trigonometric functions |
# steps to hold. This is a non-negative integer value.
For assertions, this entry specifies the number of simulation steps
that the expression must remain false before flagging an error. For
user-targets, the entry specifies the number of steps that the
expression must remain true before the target is considered covered.- Visible Variables. This is a list of data items that are visible in the
system where the objective is located and therefore are
candidates for use as variables in expressions. Double-clicking
on an item in this list will insert it at the current insertion
point into the expression under construction (in window item 2).
9.2.2 The Timer Objective Dialog
The numbers below refer to the labels in Figure 9.2.
| Figure 9.2: The dialog for inserting/modifying timer objectives. |
-
Name. The name must be unique with respect to the subsystem
where the objective resides.
- Timer. The data item from the model to be tracked as a timer.
The pull-down menu lists all data items in the context where the
objective resides.
- Start value. A real number specifying the initial value of the timer/counter.
- Step size. A real number specifying a step size such that the objective
will track intermediate steps between the start and end value of this size.
The step size should typically be set to the increment used in the model
to change the value of the tracked data item.
- End value. A real number specifying the end value of the timer/counter.
9.2.3 The Diagram Objective Dialog
The numbers below refer to the labels in Figure 9.3.
| Figure 9.3: The dialog for inserting/modifying diagram objectives. |
- Objective name. The name must unique within the subsystem
where the objective resides.
- Name of the currently chosen objective library. This is a
standard .mdl file storing a Simulink / Stateflow model or a library containing
the Simulink / Stateflow subsystems that will be wired into a controller model as
Validator diagram objectives. This file is constructed in the
usual manner using The MathWorks’ Simulink and Stateflow
editors. Note that the actual wiring is maintained by Reactis, so
the controller model being validated need not be modified at all.
- Clicking here opens a file selection dialog for choosing a different
objective library.
- System. This tree represents the structure of the library specified in
window item 2. Selecting an item in the tree indicates the
Simulink subsystem from the library that encodes the objective.
- Parameters. If the system chosen above is a masked subsystem,
the values for the mask parameters may be entered here.
- Wiring. This section enables the user to specify the wiring between
input ports of the diagram objective and the blocks in the
subsystem of the model where the objective resides. Labels on
the right represent the input ports of the diagram objective.
The menus to the left list data items from the model that may be
wired to the inputs of the objective.
When the dialog is dismissed, the wiring specified here may be
viewed from the main panel of the Reactis window by hovering
over the diagram objective. The wiring will be shown as blue
lines connecting inputs to the objective to data items in the
model being instrumented for validation.
9.3 Running Reactis Validator
After adding assertions and user-defined targets to a model, Validator may
be invoked by
selecting the Validate -> Check Assertions... menu item to determine
whether or not assertion violations can be uncovered.
Figure 9.4 contains an annotated screen shot of
the Validator launch dialog; the labeled items are described in the next
subsection. It should first be noted, however, that the
Validator launch dialog is very similar to that of the Tester launch
dialog as described in Section 8.1. This similarity is due
to the fact that conceptually, Validator works by generating test data
using the Tester test-generation algorithm and then running the tests on
the instrumented model to search for assertion violations. For this reason,
many of the features of the Validator launch screen are identical to those
of Tester. When this is the case, the descriptions below are very brief;
more detail may be found in Section 8.1.
| Figure 9.4: The launch dialog for Validator. |
9.3.1 Labeled Window Items
-
Enable/disable the preload phase. This phase
loads existing test suites (specified by the user in window
item 4)
to serve as a starting point for Validator run.
-
Enable/disable the random phase.
-
Enable/disable the targeted phase.
-
A list of .rst files
containing test suites to be preloaded.
When the check-box to the left of a filename is checked, unnecessary
steps (those that do not increase the level of coverage) will be
pruned from the preloaded test suite. When the check-box is not
checked, no pruning of the suite will occur.
-
Clicking this button invokes a file-selection dialog that enables the
user to specify an .rst file to be added to the preload list.
-
Clicking this button removes the currently selected .rst file from
the list of test suites to be preloaded.
- Number of tests in the random phase.
Because of pruning that occurs at the end of the random phase, some
tests may be eliminated entirely, leading to a smaller number of tests
at the end of the random phase than what is specified here.
-
Number of steps in each test of the random phase.
Upon completion of the random phase, unimportant steps are
pruned from the tests, so the lengths of the final tests will usually be
shorter than the length specified here.
-
Using this entry box, the user may specify the number of execution steps
to take during the targeted phase. The targeted phase uses
sophisticated strategies to guide the simulation to exercise parts of
the model not visited during the initial phases. The number in this
entry specifies an upper bound on the number of simulation steps
executed during this phase.
-
This entry box enables the user to pass one or more
of the following parameters to Validator :
-
-a {0|1} turns inputs abstraction on (1) or off (0).
Inputs abstraction usually improves the performance of Validator and
should be left on (default). In rare cases, turning it off may improve
coverage. If coverage problems are encountered with inputs abstraction
on, it may be beneficial to take a test suite produced with abstraction
on, preload
it into Validator, turn abstraction off, and then run Validator again.
-s randomSeed seed for the random number generator.
This is useful for replaying a previous run of Tester. The
random seed used to create a .rst file can be found in the test-suite
log (which may be viewed in the Test Suite Browser described
in Chapter 11), after the “-s”
in the “Created by Tester:” line.
-
The name of the .rst file
to be generated.
-
Clicking this button opens a file-selection dialog for specifying
the name of the .rst file to be generated.
- Clicking this button displays on-line Validator help.
- Clicking this button opens a file selection dialog to
specify an .rtp file
from which to load Tester launch
parameters. Reactis may be configured from the
Settings dialog to generate an .rtp file for each
Tester or Validator run.
- Reset the Validator parameters to their default values.
- Scroll backward in the parameter history.
- Scroll forward in the parameter history.
- Clicking this button starts Validator run.
- Clicking this button closes the Validator window.
The Progress Dialog displayed while Validator is running is the
same as the Progress Dialog for Tester. For more information
see Section 8.2
9.4 Validator Menus in the Reactis Top-Level Window
Chapters 4 and 7 describe most
of the menu entries of the menu bar in the Reactis Top-Level
Window. We now describe the remaining entries which are related
to Validator.
-
Edit menu.
-
This menu includes entries used to manipulate Validator
objectives which are stored in .rsi files. Note, that .rsi files
may be modified only when Simulator is disabled, therefore, when
Simulator is active, these items are disabled.
-
Undo.
- Undo an operation (add, edit, remove, move) on a Validator
objective.
- Redo.
- Redo last undone operation (add, edit, remove, move) on a Validator
objective.
- Cut.
- Cut the currently selected Validator objective.
- Copy.
- Copy the currently selected Validator objective to the clipboard.
- Paste.
- Paste a Validator objective from the clipboard
to the current subsystem. To paste an objective to a specific
position, right-click on that position in your model and select Paste from the context menu.
- Validate menu.
-
-
Add Assertion.
- Add a new Validator assertion at a default location
of the currently selected subsystem.
- Add User-Defined Target.
-
Add a new Validator target at a default location
of the currently selected subsystem.
- Edit Objective...
- Edit the currently selected objective.
- Remove Objective...
- Remove the currently selected objective.
- Check Assertions...
- Start a search for tests that violate assertions
or cover user-defined targets.
