Table 1
Example of a Script Concordance Test item featuring 3 questions nested within one case. Clinical scenario: You are evaluating a 63-year-old woman with left-sided weakness in the Emergency Department of your hospital
|
If you were thinking …: |
And then you find …: |
Your hypothesis becomes …: | ||||
|---|---|---|---|---|---|---|
|
Q1. Brain abscess |
Patient had dental work 1 month ago |
−2 |
−1 |
0 |
+1 |
+2 |
|
Q2. Stroke |
Patient uses a vaginal oestrogen cream once daily |
−2 |
−1 |
0 |
+1 |
+2 |
|
Q3. Brain metastasis |
Normal non-contrast CT head |
−2 |
−1 |
0 |
+1 |
+2 |
−2: Ruled out or almost ruled out; −1: Less likely; 0: Neither more nor less likely; +1: More likely; +2: Certain or almost certain
Table 2
Test characteristics
|
Radiation-oncology [20] |
Neurology [21] |
General surgery [22] | |
|---|---|---|---|
|
Participants | |||
|
N clerks |
70 |
8 |
0 |
|
N residents (R1–R5) |
38 |
41 |
202 |
|
N panellists |
42 |
16 |
21 |
|
Items | |||
|
N cases |
30 |
24 |
43 |
|
N questions |
70 |
79 |
131 |
|
Proportion of items with panel modal response: | |||
|
−2 |
6% |
9% |
26% |
|
−1 |
38% |
27% |
40% |
|
0 |
24% |
29% |
27% |
|
+1 |
22% |
29% |
3% |
|
+2 |
10% |
6% |
3% |
|
Cronbach’s alpha |
0.90 |
0.79 |
0.85 |
Table 3
Distribution of responses according to sub-groups of respondents
|
−2 |
−1 |
0 |
+1 |
+2 | ||
|---|---|---|---|---|---|---|
|
Radiation-oncology |
Clerks |
10% |
28% |
28% |
22% |
12% |
|
Residents |
9% |
27% |
22% |
28% |
14% | |
|
Panellists |
11% |
27% |
29% |
20% |
13% | |
|
Neurology |
Clerks |
13% |
30% |
23% |
25% |
10% |
|
Residents |
10% |
30% |
23% |
25% |
12% | |
|
Panellists |
10% |
28% |
31% |
23% |
8% | |
|
General surgery |
Residents |
19% |
38% |
23% |
14% |
6% |
|
Panellists |
25% |
34% |
23% |
11% |
7% |
Table 4
Scores of actual respondents, a hypothetical examinee providing random answers (i.e. guessing), and a hypothetical examinee using an ‘all 0’ strategy (i.e. gaming)
|
Groups |
N |
Mean |
SD |
Z score* | |
|---|---|---|---|---|---|
|
Radiation-oncology | |||||
|
Actual respondents |
Clerks |
70 |
51.6 |
8.2 | |
|
Residents |
38 |
71.2 |
9.5 | ||
|
All examinees |
108 |
59.3 |
9.7 | ||
|
Guessing |
Random 5 |
100 |
35.8 |
4.0 |
−2.42 |
|
Random 3 |
100 |
43.7 |
2.7 |
−1.63 | |
|
Gaming |
All 0 |
52.2 |
−0.73 | ||
|
Neurology | |||||
|
Actual respondents |
Clerks |
8 |
63.6 |
6.3 | |
|
Residents |
41 |
70.3 |
8.8 | ||
|
All examinees |
49 |
67.9 |
8.8 | ||
|
Guessing |
Random 5 |
100 |
35.9 |
4.4 |
−3.6 |
|
Random 3 |
100 |
50.9 |
4.1 |
−2.0 | |
|
Gaming |
All 0 |
53.2 |
−1.7 | ||
|
General surgery | |||||
|
Actual respondents |
Residents |
202 |
68.1 |
7.4 | |
|
Guessing |
Random 5 |
100 |
42.6 |
3.0 |
−3.4 |
|
Random 3 |
100 |
53.8 |
3.2 |
−1.9 | |
|
Gaming |
All 0 |
51.7 |
−2.2 | ||
*Z scores indicate by how many standard deviations a score deviates from the mean
Table 5
Effect on actual respondent scores (using data from Lambert et al. [22]) of excluding questions with a modal panel response of 0 one-by-one, and recalculating the examinee’s final score after each item was discarded. In shaded rows, a pass-fail threshold of 2 standard deviations below the mean would ensure that gamers fail the test
|
N of questions with a modal panel response of 0 |
% Questions with a modal panel response of 0 |
‘All-0’ score for a hypothetical examinee |
Mean score of actual examinees |
Min |
Max |
All 0 z score |
Cronbach’s alpha |
|---|---|---|---|---|---|---|---|
|
22 |
24.4% |
52.2 |
62.0 |
37.3 |
84.1 |
−0.90 |
0.904 |
|
21 |
23.3% |
51.0 |
61.2 |
37.3 |
83.0 |
−0.95 |
0.902 |
|
20 |
22.2% |
49.9 |
60.5 |
36.2 |
81.8 |
−1.00 |
0.900 |
|
19 |
21.1% |
48.8 |
59.7 |
35.8 |
80.7 |
−1.04 |
0.896 |
|
18 |
20.0% |
47.7 |
59.2 |
35.8 |
79.6 |
−1.12 |
0.893 |
|
17 |
18.9% |
46.6 |
58.5 |
35.8 |
78.5 |
−1.18 |
0.892 |
|
16 |
17.8% |
45.5 |
57.9 |
35.8 |
77.4 |
−1.27 |
0.891 |
|
15 |
16.7% |
44.4 |
57.4 |
35.6 |
76.3 |
−1.34 |
0.891 |
|
14 |
15.6% |
43.3 |
56.8 |
35.5 |
75.2 |
−1.42 |
0.888 |
|
13 |
14.4% |
42.2 |
56.3 |
35.4 |
74.1 |
−1.52 |
0.884 |
|
12 |
13.3% |
41.0 |
55.8 |
34.3 |
73.6 |
−1.61 |
0.882 |
|
11 |
12.2% |
39.9 |
55.1 |
34.1 |
72.7 |
−1.67 |
0.881 |
|
10 |
11.1% |
38.8 |
54.2 |
33.7 |
71.6 |
−1.72 |
0.879 |
|
9 |
10.0% |
37.7 |
53.6 |
33.9 |
70.8 |
−1.87 |
0.876 |
|
8 |
8.9% |
36.6 |
52.9 |
32.6 |
70.4 |
−1.94 |
0.875 |
|
7 |
7.8% |
35.5 |
52.3 |
31.5 |
69.3 |
−1.99 |
0.871 |
|
6 |
6.7% |
34.4 |
51.7 |
31.1 |
68.2 |
−2.08 |
0.868 |
|
5 |
5.6% |
33.3 |
51.4 |
30.4 |
68.2 |
−2.19 |
0.865 |
|
4 |
4.4% |
32.3 |
50.6 |
30.0 |
67.1 |
−2.22 |
0.865 |
|
3 |
3.3% |
31.0 |
50.1 |
30.0 |
66.8 |
−2.37 |
0.860 |
|
2 |
2.2% |
29.9 |
49.3 |
29.4 |
65.6 |
−2.43 |
0.858 |
|
1 |
1.1% |
28.8 |
48.6 |
29.2 |
65.1 |
−2.52 |
0.857 |
|
0 |
0.0% |
27.7 |
47.7 |
28.3 |
64.0 |
−2.53 |
0.859 |
Fig. 1
Z-score of a ‘gamer’ on subtests with different proportions of questions with modal panel responses of 0
