Chapter 12   Executing Tester-Generated Test Suites

Once generated, test suites may be executed in a variety of different ways. Users may:

1. Load them into Reactis Simulator and run them.
2. Load them into the Simulink simulator of The MathWorks and run them, with or without the help of a graphical tool included in the Reactis distribution.
3. Generate text files for processing by test harnesses.

This chapter describes these options in more detail.

12.1  Executing Test Suites Using Reactis Simulator

---

Figure 12.1: The Reactis Simulator window.

---

Reactis Simulator includes facilities for running Tester-generated tests on Simulink / Stateflow models. The following uses Figure 12.1 to explain the steps needed to load and run a .rst file named cruise.rst.

1. Select File -> Open Model... and use the resulting dialog to select cruise.mdl for loading.
2. Click button 2 to enable Simulator.
3. Click button 14 and use the resulting dialog to select cruise.rst for loading. The name of the file will be written in the title bar of the Simulator window, and the tests will be listed in pull-down menu (item 12), together with an entry all.
4. To run a single test:
   1. Click on the down arrow of the Inputs Source dialog (window item 12).
   2. Select the test to be run. It will appear in the Inputs Source dialog.
   3. Use buttons 3–10 to run the simulator as described in Section 7.1. The inputs for each simulation step will be drawn from the test.
5. To run all tests at once:
   1. Click on the down arrow of the Inputs Source dialog (window item 12).
   2. Select the all entry.
   3. Use buttons 3–10 to run the simulator as described in Section 7.1. The inputs for each simulation step will be drawn from the tests, beginning with Test 1. When each test is completed, the simulation is “reset” so that the next test begins in the initial configuration. Coverage information, however, is not reset.

12.2  Executing Test Suites in Simulink

Test suites generated by Reactis may also be used to guide simulations in The MathWorks' Simulink environment. To run tests in Simulink, a test suite must first be exported into a format readable by MATLAB . Test suites may be exported from Reactis Simulator (described in more detail in Chapter 7) by selecting the Test Suite -> Export menu item. For running tests in Simulink, Reactis supports three export formats which are listed in the export dialog as follows.

Format 1

Matlab text (*.m)

Format 2

Matlab binary (*.mat)

Format 3

Matlab binary for 'FromWorkspace' blocks (*.mat)

In all cases, when executed, the file initializes matrices that can be used as inputs to drive a Simulink simulation. We describe four ways to load an exported test suite into Simulink and then read input values from the tests during a simulation. The four methods are listed below along with the export file formats with which they work.

12.2.1  Executing Tests in Simulink with the runtests Utility

---

Figure 12.2: The runtests utility for running test suites in Simulink.

---

The runtests tool in the Reactis distribution can be used as follows to run Reactis-generated tests from within Simulink.

1. Ensure that the Reactis “bin” folder has been added to the MATLAB path by selecting the File -> Set Path from the top-level MATLAB window. If you used the default settings during the Reactis installation then the Reactis bin directory is
   C:\Program Files\Reactis V2007\bin
2. Start the GUI by entering runtests on the MATLAB command line. You will be presented with the window shown in Figure 12.2.
3. Use the Load Model button to specify a Simulink model to load.
4. You may now click the Show Model button to view the loaded model.
5. Use the Load Test File button to specify a .m file or a .mat file (exported by Reactis) to load.
6. Select a test in the drop-down box.
7. Click the Run Test button to execute the selected test.
8. Any differences between the model's outputs as computed by Reactis and those computed by Simulink will be listed in the box titled Differences found. A difference between outputs is listed if it exceeds the the error tolerance specified in the Eps box. More precisely, for an outport p, let
   

   eps	be the value specified in the Eps box
   v1	be the value in the test suite for outport p at step i of test j
   v2	be the value computed by Simulink for outport p at step i of test j

   
   then an error for outport p at step i of test j exceeds the specified tolerance if | v₁ − v₂ | > eps × | v₁ |

12.2.2  Executing Tests in Simulink with the Simulation Parameters Dialog

An alternative to the runtests utility is running the tests manually within MATLAB as follows.

1. Load the .m file (or .mat file) by entering the file name without the .m (.mat) file suffix on the MATLAB command line. ¹ For example:

         >> cruiseTests
   

2. Load the model in Simulink and select the Simulation -> Configuration Parameters menu item.
3. In the Solver panel of the Configuration Parameters dialog, adjust the simulation time to use the entire test. By convention, Reactis stores the start and stop times for test n, where n is the number of the test (1, 2, 3, etc.), in variables named testn_start and testn_stop, respectively.
4. In the Data Import/Export panel of the Configuration Parameters dialog, enable the Input check box and enter the name of the test matrix in the text box next to it. The test matrices in a Reactis-exported .m file (.mat file) are by convention named test1, test2, test3, ... . For example, enter test1 in the text box if you want to execute the first test.
5. Start a simulation run in Simulink. The simulation inputs will be taken from the specified test.

12.2.3  Running Tests in Simulink by Editing the .m file

This method also allows tests to be run from within MATLAB . The difference with the previous method is that it allows the “batch processing” of multiple tests.

1. Open the .m file in the MATLAB editor (by selecting File -> Open from the top-level MATLAB window).
2. In line 2 of the .m file make sure modelName is assigned the name of the model you would like to simulate.
3. Scroll to the bottom of the file and you will find a number of commented-out lines, one for each test. Uncomment the lines of tests you want to execute and save the file.
4. Load the file by entering its name on the MATLAB command line, and all uncommented tests will be executed.

12.2.4  Running Tests in Simulink by Replacing Top-Level Inports with FromWorkspace Blocks.

Executing tests exported in format 3 requires some modifications to be made to a model before the tests may be run in Simulink. For each test in a test suite, the exported file contains a variable named testn (where n is the test number). The variable stores an array of structures ready for use in a FromWorkspace block. Each element of the array stores the values that one inport assumes during the test. A model may be modified to use these structures to execute test n as follows:

1. Load the model in Simulink.
2. For each top-level inport,
   1. Delete the inport and replace it with a FromWorkspace block.
   2. Double-click on the new FromWorkspace block.
   3. In the Data field, enter testn(i) (where n is the test number and i is the inport number. For example test3(5) specifies test 3, inport 5).
   4. Uncheck the Interpolate field.
   5. Select Holding final value in the Form output after final value by field.
3. In the Solver tab of the Simulation Parameters dialog, adjust the simulation time to use the entire test. By convention, Reactis stores the start and stop times for test n, where n is the the test number, in variables named testn_start and testn_stop, respectively.
4. Start a simulation run in Simulink. The simulation inputs will be taken from the specified test.

12.3  Executing Test Suites on Source Code

Tester-generated test suites may also be run on source-code implementations of Simulink / Stateflow models; this feature helps automate the important task of determining whether deployed code will conform to models. Reactis currently supports two different methods for running tests on source.

Via Simulink:

Testers may use the S-function feature of MathWorks' Simulink environment to embed their source code inside a Simulink block. The .m file feature described in Section 12.2 above may then be used to generate MATLAB scripts from .rst files that can then be applied to the S-function, with differences between S-function outputs and outputs stored in tests noted and recorded.

Via user-written test harnesses:

Reactis is also capable of outputting test files in simple textual formats. Exported text file types include comma separated value format (files are suffixed by .csv) or or a more verbose representation (files are suffixed by .txt). In both cases test steps are listed in an easy-to-process fashion. Users may write their own test harnesses to read these files and run their program on the test data.

Both the .m files and text files may be obtained using the Reactis Simulator export facility; see Section 7.6.2 for more on how to do this.

In order to write test harnesses that process Reactis-exported text files, one must know about the format of these files. Figure 12.3 contains a partial listing of a .txt filegenerated from the cruise-control example used in Chapter 3.

---

*** TEST 1
|InitialSpeed                            0.0
*** STEP 1 INPUT
cruise|@onOff                            1.0
cruise|@accelResume                      1.0
cruise|@cancel                           0.0
cruise|@decelSet                         1.0
cruise|@brake                            0.0
cruise|@gas                              0.0
cruise|@inactiveThrottleDelta            -0.1
cruise|@drag                             -0.30393583527515
*** STEP 1 OUTPUT
___t___                                  0.0
cruise|@active                           0.0
cruise|@throttleDelta                    -0.1
*** STEP 2 INPUT
cruise|@onOff                            1.0
cruise|@accelResume                      1.0
cruise|@cancel                           0.0
cruise|@decelSet                         0.0
cruise|@brake                            1.0
cruise|@gas                              1.0
cruise|@inactiveThrottleDelta            0.0
cruise|@drag                             -0.832183313246298
*** STEP 2 OUTPUT
___t___                                  1.0
cruise|@active                           0.0
cruise|@throttleDelta                    -0.1

Figure 12.3: A sample fragment of an exported test suite (ASCII format).

---

Each .txt file contains a list of tests, with the beginning of each test indicated by a line containing the string *** TEST x, where x is the number of the test. After this line, a sequence of lines of name/value pairs appears: these are the configuration variables and the values they are assuming for the duration of the test. Following the configuration section is the sequence of steps in the test, with a step consisting of an input section followed by an output section. Each input section is introduced by a line containing *** STEP x INPUT, where x is the step number within the test; each output section is preceded by a line whose content is *** STEP x OUTPUT. Note that the step number for an output section must be the same as the immediately preceding input section. Each input / output section then consists of a sequence of variable name / value lines. Each output section also contains a special variable, ___t___, recording the simulation time of the step.

Reactis uses a special scheme to encode the names of variables in configuration, input and output sections. Because variables of the same name but at different levels in the model hierarchy are different in Simulink / Stateflow, Reactis refers to variables using hierarchy information in addition to variable names. The character . is used to separated different levels in the hierarchy, with a | delimiting the end of the system name and the beginning of the variable name. So cruise|onOff refers to the (inport) variable onOff at the top-level in the cruise model. The | preceding variable name |InitialSpeed indicates that InitialSpeed is a global variable.

Finally, non-alphanumeric characters that appear in variable or subsystem names are encoded using three-digit ASCII codes preceded by the _ symbol. Thus, a variable named eng_on in a Simulink / Stateflow model would be rendered as eng_095on. This is because the _ character is non-alphanumeric with ASCII code 95. A variable named eng revving would be rendered as eng_032revving, since the ASCII code for the space character is 32.

1

Note that if an .m file and an .mdl file have the same base name, then entering that base name on the MATLAB command line causes the .mdl file to be loaded.