Chapter 13  Troubleshooting Coverage Problems

Reactis Tester and Validator use sophisticated algorithms to generate tests that maximize coverage criteria. Nevertheless, users will sometimes encounter models that present challenges to these algorithms: generated test suites may not provide coverage adequate to the users' needs. This chapter discusses strategies for overcoming such situations.

When attempting to improve coverage, the following general strategy is often useful.

1. Load the model in Reactis and start Simulator.
2. Load a test suite.
3. Select Coverage -> Show Details and run all tests in the suite.
4. Inspect uncovered targets. Are targets coverable?

   no

   Perhaps the uncovered targets are in fact unreachable. The unreachable targets might be intended (for example, the upper or lower bound of a saturation block that you never expect to be exercised) or unintended due to a modeling error.

   yes

   If you determine that the uncovered targets could be covered, then try applying one of the techniques described in the following sections to enhance the coverage of the test suite.

13.1  Modifying the .rsi File

In many cases coverage can be improved if tighter bounds are imposed on input values in the .rsi file. Replacing range types by enumeration types can be especially handy, as can including appropriate resolution values: the goal in both cases is to reduce the number of possible values Reactis must consider when generating tests.

Probability weights may also be used to affect the likelihood with which different conditions become true. For example, in some models large portions of the model become unreachable if an input flag has a certain value (e.g. this input might indicate the system being modeled should shut down). Assigning low probabilities to those values can increase coverage. Chapter 5 describes how .rsi files can be created and modified.

When modifying the .rsi file in this fashion, it is good practice to save the modified .rsi file as a new file, so that the original settings are not overwritten. This can be done using the File -> Save Info File As... menu item in the top-level Reactis menu, as described in Chapter 5. Note that this operation automatically associates the new .rsi file with the model. The link to the previous .rsi file can be restored using the File -> Select Info File... menu item.

13.2  Modifying Test-Generation Parameters

Another approach for improving coverage is to change the parameters used by Tester and Validator. These parameters may be set in the launch screens of the respective tool and may be categorized as follows.

Preload phase parameter.

This parameter consists of a list of files of test data that Tester / Validator should load initially. When a given collection of test data provides insufficient coverage, a good strategy to try is to re-run Tester / Validator using this file of test data in the Preload phase.

Random phase parameters.

The random phase includes two parameters: the number of different tests to generate initially, and an upper bound on the number of steps in each of these tests.

In some cases, increasing the number of tests can improve coverage. For example, in some models different settings of configuration variables make parts of the model uncoverable. Increasing the number of tests in the random phase will increase the number of different configuration-variable settings that are considered. As a rule of thumb, one should set the number of tests to twice the number of distinct configuration-variable settings one wishes to consider.

Targeted phase parameter.

The Targeted phase parameter determines the number of extra simulation steps Reactis will use to try to cover parts of the model not yet covered by the other phases. Increasing this number can also improve coverage.

13.3  Test-Suite Tuning

If adjusting the test-generation parameters still does not result in a satisfactory test suite, then some hand-tuning of the suite may be necessary. Reactis Simulator and Tester / Validator contain several features designed to facilitate this process, including test-suite preloading (Tester), and test-suite execution, breakpoints and test-suite editing (Simulator).

The general strategy to be pursued may be summarized as follows.

1. Build an initial test suite.
2. Load the test suite into Simulator and add in a test that causes a “hard-to-cover” item to be covered.
3. Preload the modified test-suite into Tester / Validator and rerun.

The idea is to help Tester cover things it finds difficult while still letting Tester do as much work as possible.

Coverage problems often arise because of difficult-to-satisfy conditions on Stateflow transition segments or within Simulink. Conditions might be difficult to satisfy for several reasons.

1. A condition might become true when a timer expires, meaning that control must reside in the source state of the transition segment labeled by the condition for a number of simulation steps. If other segments' conditions become true in the meantime, then the state may be exited before the timer has a chance to expire.
2. A condition might be satisfied by a relatively small subset of the values its variables might be able to assume. An example would be a comparison of a floating-point variable to a constant: only one value for the variable makes the condition true.

The strategy for coping with these problems can be summarized as follows.

1. Identify an uncovered transition-segment whose source state has been covered by a previously constructed test suite.
2. Using the test suite and breakpoints, advance execution until the source state is active.
3. Using user-guided simulation (Section 7.3.1), perform simulation steps that cause the uncovered item to become covered.
4. Add these simulation steps into a new test, save it, and re-run Tester.

The specific steps needed to implement this strategy are as follows. It is assumed that Simulator is already running and that the earlier test suite has been loaded.

1. Click on the Coverage menu and make sure that Show Details is enabled.
2. Designate the (covered) source state of the uncovered item as a breakpoint by right-clicking on it in the hierarchy panel of Simulator and toggling the breakpoint indicator.
3. In the Inputs Source dialog governing the source of inputs, select all.
4. Run the tests by clicking the Fast Simulation (    ) button. Simulator will execute until the breakpoint is tripped. Because execution breaks immediately prior to entry into the breakpoint state, hit the step button (    ) to cause the state to be entered.
5. Select user in the Inputs Source dialog.
6. Using your knowledge of the system, select (a sequence of) input values that drive execution to cover the uncovered item.
7. From the Test Suite menu, select Add/Extend Test. This will cause a new test to be added to the current test suite, which can now be seen in the Inputs Source dialog.
8. Save the current test suite using Save / Save As in the Test Suite menu.

The newly saved test suite may now be preloaded into Tester using the Preload file-selection box (make sure the Preload phase is enabled), and Tester rerun. This strategy may be repeated until an acceptable test suite is constructed. Be sure to save the resulting test suite so that it will not accidentally be overwritten by new runs of Tester.

13.4  Model Decomposition

If the steps described above still result in unsatisfactory coverage, consider testing subsystems of your model independently. Reactis makes this possible through a facility to extract subsystems from a model. The idea is to extract a subsystem, along with portions of the model involved in triggering the subsystem, and store the results as a new model in a separate .mdl file.

The subsystem extraction utility is invoked by loading a model into Reactis, then right-clicking on the Simulink subsystem or Stateflow chart to be extracted, and selecting the “Extract Subsystem” entry in the pop-up menu. In the resulting dialog a filename under which to save the extracted subsystem may be specified. Reactis will then extract the subsystem, save it under its new name, open a new Reactis window, and load the extracted model. Note that the extracted subsystem is saved as a standard .mdl file. That makes it easy to edit the extracted model using Simulink if changes to the triggering mechanism or inputs or outputs of the model are necessary.

If the extracted subsystem is a triggered subsystem or located within a triggered subsystem or contains triggered subsystems whose triggers are connected to something outside the extracted subsystem, Reactis will keep all blocks and charts that are part of the triggering mechanism. This works recursively, i.e. if the triggering block or chart is triggered itself, Reactis will trace back that trigger, too.

To determine which subsystems might be candidates for independent testing, you can use the following strategy.

1. Generate a test suite for the full model.
2. Load the test suite into Reactis Simulator and select Coverage -> Show Details.
3. Run the entire test suite on the model. See Chapter 12 for an explanation of how to do this.
4. Inspect the subsystems within the model to look at the coverage obtained. Subsystems having poor coverage are good candidates for being tested independently.