Chapter 12   Executing Tester-Generated Test Suites

Once created with Reactis, test suites may be executed in a variety of different ways. Users may:

1. Load them into Reactis Simulator and run them.
2. Load them into the Simulink simulator of The MathWorks and run them, with or without the help of the graphical runtests tool included in the Reactis distribution.
3. Generate text files for processing by test harnesses.

This chapter describes these options in more detail.

12.1  Executing Test Suites Using Reactis Simulator

---

Figure 12.1: The Reactis Simulator window.

---

Reactis Simulator includes facilities for running Tester-generated tests on Simulink / Stateflow models. The following uses Figure 12.1 to explain the steps needed to load and run a .rst file named cruise.rst.

1. Select File -> Open Model... and use the resulting dialog to select cruise.mdl for loading.
2. Click button 2 to enable Simulator.
3. Click button 14 and use the resulting dialog to select cruise.rst for loading. The name of the file will be written in the title bar of the Simulator window, and the tests will be listed in Source-of-Inputs dialog (item 12), together with an entry all.
4. To run a single test:
   1. Click on the Source-of-Inputs dialog button (window item 12).
   2. Select the test to be run. It will appear in the Source-of-Inputs dialog.
   3. Use buttons 3–10 to run the simulator as described in Section 7.1. The inputs for each simulation step will be drawn from the test and the values computed for test points and top-level outputs will be compared against those stored in the test suite.
5. To run all tests at once:
   1. Click on the Source-of-Inputs dialog button (window item 12).
   2. Click the All button.
   3. Use buttons 3–10 to run the simulator as described in Section 7.1. The inputs for each simulation step will be drawn from the tests (beginning with Test 1) and the values computed for test points and top-level outputs will be compared against those stored in the test suite. When each test is completed, the simulation is “reset” so that the next test begins in the initial configuration. Coverage information, however, is not reset.

12.2  Executing Test Suites in Simulink

Test suites generated by Reactis may also be used to guide simulations in The MathWorks’ Simulink environment. To run tests in Simulink, a test suite must first be exported into a format readable by MATLAB . Test suites may be exported from Reactis Simulator (described in more detail in Chapter 7) by selecting the Test Suite -> Export menu item. For running tests in Simulink, Reactis supports three export formats which are listed in the export dialog as follows.

Format 1

MATLAB script (.m)

Format 2

MATLAB binary data (.mat)

Format 3

MATLAB binary data for ’FromWorkspace’ blocks (.mat)

In all cases, when executed, the file initializes matrices that can be used as inputs to drive a Simulink simulation. We describe four ways to load an exported test suite into Simulink and then read input values from the tests during a simulation. The four methods are listed below along with the export file formats with which they work.

12.2.1  Executing Tests in Simulink with the runtests Utility

---

Figure 12.2: The runtests utility for running test suites in Simulink.

---

The runtests tool in the Reactis distribution can be used as follows to run Reactis-generated tests from within Simulink:

1. Ensure that the Reactis “bin” folder has been added to the MATLAB path by selecting the File -> Set Path from the top-level MATLAB window. If you used the default settings during the Reactis installation then the Reactis bin directory is
   C:\Program Files\Reactis V2010\bin
2. Start the GUI by entering runtests on the MATLAB command line. You will be presented with the window shown in Figure 12.2.
3. Use the Load Model button to specify a Simulink model to load.
4. You may now click the Show Model button to view the loaded model.
5. Use the Load Test File button to specify a .m file or a .mat file (exported by Reactis) to load.
6. Select a test in the drop-down box.
7. Click the Run Test button to execute the selected test.
8. Click the Save Log button to save a copy of the output screen to a user-selected text file.
9. Any differences between the model’s outputs as computed by Reactis and those computed by Simulink will be listed in the box titled Differences found. A difference between outputs is listed if it exceeds the error tolerance specified in the Eps box. More precisely, for an outport p, let
   

   eps	be the value specified in the Eps box
   v1	be the value in the test suite for outport p at step i of test j
   v2	be the value computed by Simulink for outport p at step i of test j

   
   then an error for outport p at step i of test j exceeds the specified tolerance if | v₁ − v₂ | > eps × | v₁ |

12.2.2  Executing Tests in Simulink with the Simulation Parameters Dialog

An alternative to the runtests utility is running the tests manually within MATLAB as follows.

1. Load the .m file (or .mat file) by entering the file name without the .m (.mat) file suffix on the MATLAB command line. ¹ For example:

         >> cruiseTests
   

2. Load the model in Simulink and select the Simulation -> Configuration Parameters menu item.
3. In the Solver panel of the Configuration Parameters dialog, adjust the simulation time to use the entire test. By convention, Reactis stores the start and stop times for test n, where n is the number of the test (1, 2, 3, etc.), in variables named testn_start and testn_stop, respectively.
4. In the Data Import/Export panel of the Configuration Parameters dialog, enable the Input check box and enter the name of the test matrix in the text box next to it. The test matrices in a Reactis-exported .m file (.mat file) are by convention named test1, test2, test3, ... . For example, enter test1 in the text box if you want to execute the first test.
5. Start a simulation run in Simulink. The simulation inputs will be taken from the specified test.

12.2.3  Running Tests in Simulink by Editing the .m file

This method also allows tests to be run from within MATLAB . The difference with the previous method is that it allows the “batch processing” of multiple tests.

1. Open the .m file in the MATLAB editor (by selecting File -> Open from the top-level MATLAB window).
2. In line 2 of the .m file make sure modelName is assigned the name of the model you would like to simulate.
3. Scroll to the bottom of the file and you will find a number of commented-out lines, one for each test. Uncomment the lines of tests you want to execute and save the file.
4. Load the file by entering its name on the MATLAB command line, and all uncommented tests will be executed.

12.2.4  Running Tests in Simulink by Replacing Top-Level Inports with FromWorkspace Blocks.

Executing tests exported in format 3 requires some modifications to be made to a model before the tests may be run in Simulink. For each test in a test suite, the exported file contains a variable named testn (where n is the test number). The variable stores an array of structures ready for use in a FromWorkspace block. Each element of the array stores the values that one inport assumes during the test. A model may be modified to use these structures to execute test n as follows:

1. Load the model in Simulink.
2. For each top-level inport,
   1. Delete the inport and replace it with a FromWorkspace block.
   2. Double-click on the new FromWorkspace block.
   3. In the Data field, enter testn(i) (where n is the test number and i is the inport number. For example test3(5) specifies test 3, inport 5).
   4. Uncheck the Interpolate field.
   5. Select Holding final value in the Form output after final value by field.
3. In the Solver tab of the Simulation Parameters dialog, adjust the simulation time to use the entire test. By convention, Reactis stores the start and stop times for test n, where n is the test number, in variables named testn_start and testn_stop, respectively.
4. Start a simulation run in Simulink. The simulation inputs will be taken from the specified test.

12.3  Executing Test Suites on Source Code

Tester-generated test suites may also be run on source-code implementations of Simulink / Stateflow models; this feature helps automate the important task of determining whether deployed code will conform to models. Reactis currently supports several methods for running tests on source.

Via Reactis Simulator

You may use the S-function feature of MathWorks’ Simulink environment to embed your source code inside a Simulink block. This wrapper model may then be loaded and run in Simulator with differences between S-function outputs and outputs stored in tests flagged. This is especially useful when using the Reactis for C Plugin, since it allows inspection of the C code as the tests execute.

Via Simulink:

You may use the S-function feature of MathWorks’ Simulink environment to embed your source code inside a Simulink block. The .m file feature described in Section 12.2 above may then be used to generate MATLAB scripts from .rst files that can then be applied to the S-function, with differences between S-function outputs and outputs stored in tests noted and recorded.

Via user-written test harnesses:

Reactis is also capable of outputting a test suite as a comma separated value (CSV) file for easy processing by external tools. Test steps are listed in an easy-to-process fashion. Users may write their own test harnesses to read these files, run their program on the test data, and compare the outputs of the program to those stored in the tests.

Both the .m files and CSV files may be obtained using the Reactis Simulator export facility; see Section 7.6.2 for more on how to do this.

In order to write test harnesses that process Reactis-exported CSV files, one must know about the format of these files. Figure 12.3 contains a partial listing of a .csv file generated from the cruise-control example used in Chapter 3. See Section 7.6.2 for a description of the CSV file format.

---

onOff,accelResume,cancel,decelSet,brake,gas,inacThrDelt,drag,___t___,active,thrDelt,speed,"|InitialSpeed"
0.0,1.0,1.0,1.0,0.0,1.0,-0.1,-0.01,0.0,0.0,0.0,0.0,0.0
1.0,1.0,0.0,1.0,1.0,1.0,0.0,-0.00959914797712953,1.0,0.0,0.0,0.0,0.0
0.0,1.0,0.0,0.0,1.0,0.0,0.1,-0.00866942376836091,2.0,0.0,0.0,0.0,0.0
1.0,1.0,1.0,1.0,1.0,0.0,0.1,-0.00829318652470756,3.0,0.0,0.0,0.191330576231639,0.0

1.0,1.0,1.0,0.0,0.0,1.0,0.1,0.01,0.0,0.0,0.0,50.0,50.0
0.0,1.0,0.0,1.0,1.0,0.0,0.0,0.00942639853457365,1.0,0.0,0.0,50.21,50.0
0.0,0.0,0.0,1.0,1.0,1.0,-0.1,0.00884905410912495,2.0,0.0,0.0,50.2194263985346,50.0

0.0,0.0,1.0,1.0,0.0,1.0,0.1,-0.01,0.0,0.0,0.0,4.21309604376837,4.21309604376837
0.0,1.0,1.0,1.0,0.0,1.0,-0.1,-0.00955227172719837,1.0,0.0,0.0,4.40309604376837,4.21309604376837
1.0,1.0,1.0,0.0,0.0,0.0,0.1,-0.00866896857231075,2.0,0.0,0.0,4.19354377204118,4.21309604376837
0.0,1.0,1.0,1.0,0.0,0.0,0.1,-0.00855659477502143,3.0,0.0,0.0,4.38487480346887,4.21309604376837
1.0,0.0,1.0,0.0,0.0,0.0,-0.1,-0.00773304191244197,4.0,0.0,0.0,4.57631820869384,4.21309604376837
0.0,0.0,1.0,1.0,0.0,1.0,0.1,-0.00700061209826044,5.0,0.0,0.0,4.3685851667814,4.21309604376837

0.0,1.0,0.0,1.0,0.0,1.0,0.1,0.0,0.0,0.0,0.0,22.8683123376438,22.8683123376438

Figure 12.3: A sample fragment of an exported test suite (CSV format).

---

1

Note that if an .m file and an .mdl file have the same base name, then entering that base name on the MATLAB command line causes the .mdl file to be loaded.