Reactis Validator searches for defects and inconsistencies in models. The
tool enables you to formulate a property that model behavior should have as an
assertion, attach the assertion to a model, and perform an automated search for
a violation of the assertion. If Validator finds an assertion violation, it
returns a test that leads to the problem. This test may then be executed in
Reactis Simulator to gain an understanding of the sequence of events that
leads to the problem.
Validator also allows users to specify specific test scenarios they want
exercised using one of two alternative mechanisms. User-defined targets
enable you to specify an abstract test scenario, whereas virtual sources
give you a way to easily specify a concrete scenario.
User-defined targets may be seen as user-defined extensions to the built-in
coverage criteria supported by Reactis: in generating test data,
Validator (and Tester) will attempt to generate test runs that satisfy the
indicated scenarios. Like assertions, user-defined targets are attached to
models.
Virtual sources are placed at the top level of a model to control one or more
top-level inports as a model executes in Reactis. That is you can specify a
sequence of values to be consumed by an inport during simulation or
test-generation. Virtual sources can be easily enabled and disabled. When enabled,
the virtual source controls a set of inports and while disabled those inports
are treated by Reactis just as normal top-level inports.
Validator is particularly useful in requirements validation. Given a list
of requirements on model behavior, you can formulate assertions (to check
whether a requirement is being satisfied) and user-defined targets or virtual
sources (to define test scenarios intended to “stress” the requirement).
Engineers use Validator as follows. First a model is instrumented with
assertions to be checked, user-defined targets, and virtual sources.
We refer to assertions, user-defined targets, and virtual sources
as Validator objectives.
The tool is then invoked on the instrumented model to
search for assertion violations and paths leading to the specified
coverage targets. The output of a Validator run is a test suite
that includes tests leading to objectives found during the analysis.
Assertions and user-defined targets may be added to any Simulink system or
Stateflow diagram in a model; whereas, virtual sources may only be added
at the top-level of the model. When adding validator objectives to a model,
three mechanisms (only the first two are available for virtual sources)
for formulating objectives are supported:
-
Expression objectives
- are C-like Boolean expressions.
- Diagram objectives
- are references to subsystems in standard
Simulink/Stateflow libraries.
- Timer objectives
- are directives that tag a model data item
as a timer or counter.
An expression objective is a C-like Boolean expression whose
variables are wired to data items from the context of the model in which the
objective is attached. Expression objectives are easily attached and
modified from the Reactis GUI. For more information on valid
expressions see Section 9.3.1.
Diagram objectives give users the full power of Simulink / Stateflow to formulate
assertions, user-defined targets, and virtual sources. The objectives
may use any Simulink blocks supported by Reactis, including
Stateflow diagrams. Diagram objectives are attached to a model using the Reactis
GUI to select a Simulink system from a library and “wire” it into the
model. The diagrams are created using Simulink and Stateflow in
the same way standard models are built. After adding a diagram
objective to a model, the diagram is included in the model’s
hierarchy tree, as are other library links in a model.
A timer objective tags a model variable as a timer or counter. In the
case of a timer user-target, the target is considered covered after
the data item changes from its start value to its end value by a
specified increment. In the case of a timer assertion, the assertion
is considered violated if the data item ever changes from its start
value to the end value by a specified increment.
9.1 The Meaning of Validator Objectives
We now explain in more detail how Reactis interprets objectives.
Coverage of these objectives may be tracked in the standard manner,
namely through highlighting in the main Reactis panel or in the
Coverage-Report Browser. Also, in Simulator, hovering over a covered
objective shows the test and step number when it was first covered.
9.1.1 Assertions
Assertions specify properties that should always be true for a model.
For a given assertion, Validator searches a model for a simulation
run that leads to a configuration in which the assertion does not
hold. An expression assertion fails to hold if the given expression
evaluates to false1. A timer
assertion fails to hold if the data item it tracks ever changes
from its start value to the end value by a specified increment. A
diagram assertion fails to hold if any output port assumes the boolean
value “false” or a numeric value zero. Note that if a diagram has
more than one output port, the individual port names are listed in the
target list and highlighted independently according to whether they
are violated or not.
An assertion is considered “covered” when a violation is
found. So in contrast to targets, where being covered is considered
good, covering an assertion is bad. We therefore highlight covered assertions in red and yellow, in contrast to targets where
uncovered targets are highlighted in red.
9.1.2 User-Defined Coverage Targets
User-defined coverage targets extend the Reactis built-in targets
described in Chapter 6 (branch coverage, state
coverage, condition-action coverage, MC/DC, etc). If a user-defined
target is specified as an expression, Reactis will treat it as
covered when the expression evaluates to a non-zero numeric value.
If a user-defined target is specified as a timer, the target is
considered covered after the data item tracked by the timer changes
from its start value to its end value by its specified increment.
If a user-defined target is specified as a diagram, Reactis will
treat it as covered when all output ports of the associated
subsystem have assumed either a boolean “true” value or any non-zero
numeric value. Note that if a diagram has more than one output port,
the individual port names are listed in the target list and
highlighted independently according to whether they are covered or
not.
9.1.3 Virtual Sources
Virtual sources control top-level inports of a model. If a virtual source is an
expression, then at each simulation step, the expression is evaluated to compute
a value that will be fed into the top-level inport controlled by the virtual
source. If a virtual source is specified as a diagram, then each outport of
the diagram may be fed into a top-level inport of the model. Some outports
of the virtual source diagram may be left unconnected and monitored by an
assertion.
9.2 Use Cases of Validator Objectives
9.2.1 Checking a Requirement with an Expression Assertion and an Expression User-Defined Target
Consider a cruise control application that has the following requirement:
The cruise control shall remain inactive if the speed of the vehicle is less than 30 mph.
To properly test this requirement we need to execute a test in which an
attempt is made to activate the cruise control while the speed of the
vehicle is less than 30 mph. We can capture this as a simple expression-based
user-defined target:
on && activate && (speed < 30)
This expression would be "wired-in" such that on monitors whether the cruise
control is turned on, activate monitors whether an attempt has been made
to activate the cruise control, and speed is the speed of the vehicle.
When this expression becomes true, we have found a test to test our requirement.
We can test whether the applications response to this test is correct with the
following expression assertion:
!(active && (speed < 30))
This expression is wired in so that active monitors whether the cruise control
is active. If this expression ever becomes false, then the requirement is violated.
9.2.2 Checking a Requirement with a Diagram Assertion
Consider another requirement for a cruise control:
When active, cruise control shall not permit actual, desired speeds to differ by more than 1 mph for more than 3 seconds.
This requirement can be checked by a diagram assertion that monitors the vehicle
speed, whether the cruise control is active, and the speed at which the driver has
set the cruise control. The top-level interface of the Simulink subsystem (in this
case a Stateflow diagram) is shown in Figure 9.1.
| Figure 9.1: The diagram to check the speed maintenance requirement monitors three
data items in the model and raises a flag on its output if the requirement is violated. |
The simple Stateflow diagram that implements the assertion is shown in Figure 9.2.
It works as follows:
-
Initially the diagram is in the Inactive state.
- When the cruise control becomes active it enters state Active and child
state OkDiff.
- When in OkDiff, the diagram computes the difference between the vehicle
speed and the speed at which the driver has set the cruise control.
-
If the difference exceeds the tolerance, then go to state BigDiff.
- If the difference is less than the tolerance, then stay in OkDiff.
- When entering BigDiff, start a counter
-
If the speed difference is corrected within 3 steps, then return to OkDiff
- If the speed difference is not corrected within 3 steps, then flag an error
by outputting the value 0 on its outport.
| Figure 9.2: The implementation of the speed maintenance assertion. |
9.2.3 Creating a Functional Test Using a Virtual Source
Virtual sources offer a convenient mechanism for capturing functional tests.
Consider the need to test that the cruise control activates and deactivates as
expected when executing the following scenario:
| | Input | Expected Output of Active |
| 1) | Turn cruise control on | Off |
| 2) | Press set | On |
| 3) | Press cancel | Off |
| 4) | Press set | On |
| 5) | Press brake | Off |
| 6) | Press resume | On |
| 7) | Press gas | Off |
This scenario can be captured using the virtual source shown in
Figure 9.3. The Signal Builder block shown implements
the virtual source by capturing the scenario described above. The
first six signals are fed into inputs of the model while the final
output is monitored by simple assertion that compares the expected
value for active against the actual value produced by the model.
| Figure 9.3: Virtual sources offer an easy way to implement functional tests.
Any Simulink / Stateflow construct supported by Reactis may be used to implement the
virtual source. Signal Builder blocks are especially useful. A Validator
assertion can monitor an output of the virtual source, relieving the user of
the tedious task of checking expected values against actual responses of the model. |
9.3 Adding, Editing, and Removing Objectives
Adding, editing, and removing of objectives is only possible when
Simulator is disabled. This is necessary because objectives
are linked into a model when Simulator is invoked. However,
operations with no semantic effect, such as moving or resizing
objectives, are possible even when Simulator is active.
A Validator objective may be added to a model by first selecting
in the hierarchy panel the Simulink subsystem or Stateflow
state in which the objective should be inserted and then:
-
right-clicking in an empty space in the subsystem in the
main panel of the Reactis window and selecting
Add Assertion or Add User-Defined Target
or Add Virtual Source;
or
- by selecting the Validate -> Add Assertion or
Validate -> Add User-Defined Target or
Validate -> Add Virtual Source menu items from
the main menu bar.
Usage of the resulting dialogs is explained in Section 9.3.1
(expression objectives), Section 9.3.2 (timer objectives),
and Section 9.3.3 (diagram objectives).
In Stateflow charts, only expression objectives are
supported. Using a group of radio buttons in the
wiring dialog, one can choose for the objective to be evaluated at state entry
or exit,
or every time the chart is triggered while the state is active. These
correspond to the usual entry, exit, and during actions of
Stateflow. Evaluation of the objective occurs at the end of the
corresponding state actions (entry, during, exit).
To edit properties of an objective, either right-click on the objective and
select Edit -> Properties..., or click on the objective to select it and
then choose the Validate -> Edit Objective... menu item. Note that all the
controls in the properties dialog are disabled if Simulator is currently
active. To remove an objective, either:
-
right-click on the objective and select Remove, or
- click on the objective to select it and then choose the
Validate -> Remove Objective menu item, or
- click on the objective to select it and then press the delete key.
Standard cut, copy, and paste operations are available for objectives both from
the top-level Edit menu and by right-clicking on an objective.
Edit -> Undo and Edit -> Redo unable you to undo and redo these
operations. Objectives may be moved by dragging them with the mouse.
9.3.1 The Expression Objective Dialog
| Figure 9.4: The dialog for editing expression objectives. |
The numbers below refer to the labels in Figure 9.4.
-
Objective name. The name must be unique with respect to the subsystem
where the objective resides.
- Enable or disable the objective. When disabled:
-
assertions are not checked
- user-defined targets are not tracked
- virtual sources do not control inports
- Expression. Reactis evaluates this expression at each simulation
step and interprets the scalar result of numeric type as follows:
-
Assertions.
- A zero result means a fail has occurred.
- User-Defined Targets.
- A non-zero result means the target is covered.
- Virtual Sources.
- The result is fed into the inport controlled by the virtual source.
Free variables in the expression are declared in the Inputs
section (window item 5) and wired to data items
(Simulink blocks or Stateflow variables) visible at the
place in the model where the objective is attached.
Valid operators and functions are listed in Table 9.1.
| Symbol/Name | Operation |
| +, -, *, / | Add, Subtract, Multiply, Divide |
| <, > | Less than, Greater than |
| <=, >= | Less than or equal, Greater than or equal |
| == | Equal |
!=, ~= | Not equal |
| ! | Logical Not |
| |, || | Logical Or |
| &, && | Logical And |
| [] | Vector element access |
| abs(), fabs() | Absolute value |
x^y, pow(x,y), power(x,y) | x to the power y |
| exp() | Exponent |
| ln(), log(), log10() | Logarithm |
| sqrt() | Square root |
| rem() | Division remainder |
| inf | Infinity value |
| floor(), ceil() | Rounding |
| cos(), sin(), tan() | Trigonometric functions |
| cosh(), sinh(), tanh() | Hyperbolic trigonometric functions |
| acos(), asin(), atan() | Inverse trigonometric functions |
| Table 9.1: Expression objective operators and functions. |
# steps to hold. This is a non-negative integer value.
For assertions, this entry specifies the number of simulation steps
that the expression must remain false before flagging an error. For
user-targets, the entry specifies the number of steps that the
expression must remain true before the target is considered covered.- Inputs. This is where variables used to construct the expression are
declared. Clicking the Add Variable button adds a row to the section.
The rightmost entry box of a row is where you enter the name of a variable. The
column(s) to the left of the variable name specify which data item from the
model is wired to the variable. Clicking the X button to the right
of a row deletes the variable declaration.
When the dialog is dismissed, the wiring specified here may be viewed from the
main panel of the Reactis window by hovering over the objective. The
wiring will be shown as blue lines connecting inputs to the objective to data
items in the model being instrumented for validation.
- Auto-Wire. Given an input name, clicking on this button automatically selects
the values for the left columns. Auto-wiring only works when the input name can
be uniquely resolved. If there are two or more possible matches for a name,
the auto-wiring attempt will fail and the name will need to be manually bound to
an input.
- Output. This portion of the dialog is only present when creating virtual
sources. The pull-down menu enables you to specify the model inport that the
virtual source controls.
9.3.2 The Timer Objective Dialog
The numbers below refer to the labels in Figure 9.5.
| Figure 9.5: The dialog for inserting/modifying timer objectives. |
-
Name. The name must be unique with respect to the subsystem
where the objective resides.
- Timer. The data item from the model to be tracked as a timer.
The pull-down menu lists all data items in the context where the
objective resides.
- Enable or disable the objective. When disabled the timer target is not tracked.
- Start value. A real number specifying the initial value of the timer/counter.
- Step size. A real number specifying a step size such that the objective
will track intermediate steps between the start and end value of this size.
The step size should typically be set to the increment used in the model
to change the value of the tracked data item.
- End value. A real number specifying the end value of the timer/counter.
9.3.3 The Diagram Objective Dialog
The numbers below refer to the labels in Figure 9.6.
| Figure 9.6: The dialog for inserting/modifying diagram objectives. |
- Objective name. The name must be unique within the subsystem
where the objective resides.
- Enable or disable the objective. When disabled:
-
assertions are not checked
- user-defined targets are not tracked
- virtual sources do not control inports
- Name of the currently chosen objective library. This is a standard
.mdl file storing a Simulink library containing the Simulink / Stateflow subsystems that
will be wired into a controller model as Validator diagram objectives. This
file is constructed in the usual manner using The MathWorks’ Simulink and
Stateflow editors. Note that the actual wiring is established and maintained
by Reactis, so the controller model being validated need not be modified at
all.
- Clicking here opens a file selection dialog for choosing a different
objective library.
- System. This tree represents the structure of the library specified in
window item 2. Selecting an item in the tree indicates the
Simulink subsystem from the library that encodes the objective.
- Parameters. If the system chosen above is a masked subsystem,
the values for the mask parameters may be entered here.
- Inputs. This section enables the user to specify the wiring between
input ports of the diagram objective and the blocks in the
subsystem of the model where the objective resides. Labels on
the right represent the input ports of the diagram objective.
The menus to the left list data items from the model that may be
wired to the inputs of the objective.
When the dialog is dismissed, the wiring specified here may be
viewed from the main panel of the Reactis window by hovering
over the diagram objective. The wiring will be shown as blue
lines connecting inputs of the objective to data items in the
model being instrumented for validation.
- Outputs. This portion of the dialog is only present when creating virtual
sources. The pull-down menus enable you to specify the model inports that the
virtual source controls. Each output of the virtual source may control one
input of the model. Virtual source outputs left unconnected may be monitored by
assertions.
When the dialog is dismissed, the wiring specified here may be
viewed from the main panel of the Reactis window by hovering
over the diagram objective. The wiring will be shown as blue
lines connecting each outport of the virtual source to the model
inport that it controls. Alternatively, you may hover over a
controlled model inport to display the blue line connected to
the virtual source that controls it.
9.4 Running Reactis Validator
After adding assertions and user-defined targets to a model, Validator may
be invoked by
selecting the Validate -> Check Assertions... menu item to determine
whether or not assertion violations can be found.
Figure 9.7 contains an annotated screen shot of
the Validator launch dialog; the labeled items are described in the next
subsection. It should first be noted, however, that the
Validator launch dialog is very similar to that of the Tester launch
dialog as described in Section 8.1. This similarity is due
to the fact that conceptually, Validator works by generating test data
using the Tester test-generation algorithm and then running the tests on
the instrumented model to search for assertion violations. For this reason,
many of the features of the Validator launch screen are identical to those
of Tester. When this is the case, the descriptions below are very brief;
more detail may be found in Section 8.1.
| Figure 9.7: The launch dialog for Validator. |
9.4.1 Labeled Window Items
-
Enable/disable the preload phase. This phase
loads existing test suites (specified by the user in window
item 4)
to serve as a starting point for Validator run.
-
Enable/disable the random phase.
-
Enable/disable the targeted phase.
-
A list of .rst files
containing test suites to be preloaded.
- When the Prune check-box to the right of a filename is checked,
unnecessary steps (those that do not increase the level of coverage) will
be pruned from the preloaded test suite. When the check-box is not
checked, no pruning of the suite will occur.
- If the Use Virtual Sources check-box is checked for an .rst file,
then when executing the tests in the preloaded suite, values produced by
enabled virtual sources will be used for controlled inputs instead of the
input values from the test suite for those inputs. When this item is not
checked, input values from the test suite will be used for all inputs.
-
Clicking this button invokes a file-selection dialog that enables the
user to specify an .rst file to be added to the preload list.
-
Clicking this button removes the currently selected .rst file from
the list of test suites to be preloaded.
- Number of tests in the random phase.
Because of pruning that occurs at the end of the random phase, some
tests may be eliminated entirely, leading to a smaller number of tests
at the end of the random phase than what is specified here.
-
Number of steps in each test of the random phase.
Upon completion of the random phase, unimportant steps are
pruned from the tests, so the lengths of the final tests will usually be
shorter than the length specified here.
-
Using this entry box, the user may specify the number of execution steps
to take during the targeted phase. The targeted phase uses
sophisticated strategies to guide the simulation to exercise parts of
the model not visited during the initial phases. The number in this
entry specifies an upper bound on the number of simulation steps
executed during this phase.
-
This entry box enables the user to pass one or more
of the following parameters to Validator :
-
-a {0|1} turns inputs abstraction on (1) or off (0).
Inputs abstraction usually improves the performance of Validator and
should be left on (default). In rare cases, turning it off may improve
coverage. If coverage problems are encountered with inputs abstraction
on, it may be beneficial to take a test suite produced with abstraction
on, preload
it into Validator, turn abstraction off, and then run Validator again.
-s randomSeed seed for the random number generator.
This is useful for replaying a previous run of Tester. The
random seed used to create a .rst file can be found in the test-suite
log (which may be viewed in the Test Suite Browser described
in Chapter 11), after the “-s”
in the “Created by Tester:” line.
-
The name of the .rst file
to be generated.
-
Clicking this button opens a file-selection dialog for specifying
the name of the .rst file to be generated.
- Clicking this button displays on-line Validator help.
- Clicking this button opens a file selection dialog to
specify an .rtp file
from which to load Tester launch
parameters. Reactis may be configured from the
Settings dialog to generate an .rtp file for each
Tester or Validator run.
- Reset the Validator parameters to their default values.
- Scroll backward in the parameter history.
- Scroll forward in the parameter history.
- Clicking this button starts Validator run.
- Clicking this button closes the Validator window.
The Progress Dialog displayed while Validator is running is the
same as the Progress Dialog for Tester. For more information
see Section 8.2
9.5 Validator Menus in the Reactis Top-Level Window
Chapters 4 and 7 describe most of the menu
entries of the menu bar in the Reactis Top-Level Window. We now describe the
entries which are related to Validator.
-
Edit menu.
-
This menu includes entries used to manipulate Validator
objectives which are stored in .rsi files. Note, that .rsi files
may be modified only when Simulator is disabled, therefore, when
Simulator is active, these items are disabled.
-
Undo.
- Undo an operation (add, edit, remove, move) on a Validator
objective.
- Redo.
- Redo last undone operation (add, edit, remove, move) on a Validator
objective.
- Cut.
- Cut the currently selected Validator objective.
- Copy.
- Copy the currently selected Validator objective to the clipboard.
- Paste.
- Paste a Validator objective from the clipboard
to the current subsystem. To paste an objective to a specific
position, right-click on that position in your model and select Paste from the context menu.
- Validate menu.
-
The menu items include:
-
Add Assertion.
-
Add a new Validator assertion at a default location
of the currently selected subsystem.
- Add User-Defined Target.
-
Add a new Validator target at a default location
of the currently selected subsystem.
- Add Virtual Source.
- Add a new virtual source at a default location of
the currently selected subsystem.
- Edit Objective...
- Edit the currently selected objective.
- Remove Objective.
- Remove the currently selected objective.
- Enable/Disable Objective.
- Enable or Disable the objective.
- Check Assertions...
- Start a search for tests that violate assertions
or cover user-defined targets.
