Figure 1.
Schematic of the SciUnit framework. Models can be tested against experimental observations using specific tests. These tests incorporate an experimental observation and interface with the model through capabilities. Tests can be grouped into so-called test suites. The execution of a test produces a score, which describes how well the model captures the experimental observations. SciUnit also provides methods to visualize the resulting score(s), for example, in a table.
Figure 2.
Schematic of the proposed framework highlighting the three main functions: (a) overview of experimental observations; (b) validation of computational models; (c) creation of a predictions database. At its core lies the SciUnit module, which provides the infrastructure for the automated validation of the computational models. In particular, through a set of suitable tests, the computational models can be compared against experimental observations queried from the experimental database. Another set of tests, the so-called prediction tests, are then employed to extract predictions from the computational models, thus populating the predictions database.
Table 1.
Summary of ASSR deficits in schizophrenic patients in the three studies considered here
| Fundamental | Harmonic | Subharmonic | |||
|---|---|---|---|---|---|
| Drive | 40 Hz | 30 Hz | 20 Hz | 20 Hz | 40 Hz |
| Kwon et al. | ↓ | – | – | – | – |
| Vierling-Claassen et al. | ↓ | – | ↑ | ↓ | ↑ |
| Krishnan et al. | ↓ | – | – | – | – |
[i] Note. ↓ = significantly lower in patients; ↑ = significantly higher in patients; – = no significant difference between controls and patients. The tests included in the ASSRUnit module are based on this table. Krishnan et al. (2009) tested more driving frequencies than the ones shown in the table. The table only shows measures that are common to all three studies.
Figure 3.
Display all studies and all observations included in the database.
Figure 4.
Overview of the observations in the experimental literature. The command experimental_overview prints a table summarizing the results for all studies and all observations in the database. Note that by default, the qualitative study results are presented. This can be changed to the quantitative results setting the parameter entrytype to Full.
Figure 5.
By setting the meta flag to True, additional information on the studies is displayed.
Figure 6.
The experimental_overview command allows for querying for specific studies and observations using the names retrieved with the get_studies and get_observations commands.
Figure 7.
Contrasting the results of comparing two models against experimental observations. First, the model instances are created and the parameters for the control network and the schizophrenia-like network are passed on together with a name. Then, appropriate tests are created and experimental observations are passed on. In this particular example, the observation is passed on as a “ratio,” which means that the value of the output of the schizophrenia-like simulation is divided by the value of the output of the control simulation. Afterward, the tests are grouped together to form a test suite, and the two example models are run against the test suite. The results of this run are stored in the matrix score_matrix, and by evoking the view method of the SciUnit score matrix, a comparison table is shown displaying the performance of each model against each test. Note that in this example, the two models and their resulting performance are purely hypothetical and do not reflect any actual model, and furthermore, the experimental observations do not reflect any actual findings.
Figure 8.
Generating additional data. First, model instances are created and the produce_XY_plus method is used to run the simulation. The additional seed parameter contains a list of 20 RNG seeds, and a simulation is executed for each seed in that list. Thus, each simulation differs in background noise. The produce_XY_plus methods return the mean values of the outputs for the simulation runs (mcontrol4040 and mschiz4040 above), which can be used analoguously to the output of the standard produce_XY methods. However, the values of the output of each single simulation run are returned for each run (control4040 and schiz4040 above) and can then be visualized or further analyzed statistically. Note that the model parameters used in this example are not based on any actual experimental findings in schizophrenic patients and that they do not aim to reproduce any experimental observations; they are only used for demonstration purposes (for model parameters of this model that reproduce experimental observations, see the original article by Vierling-Claassen et al. [2008]).
