Table 1
Overview of the format of in-training assessment forms used
|
Baseline |
Intervention 1 |
Intervention 2 | |
|---|---|---|---|
|
Overall scoring |
5‑point scale |
Pass-fail | |
|
Comment boxes |
At the end |
At the beginning | |
|
Number of comment boxes |
1 |
4 |
2 |
|
Number of specific checklist items |
8 |
27 |
7 |
|
Number of specific rating scale items (number of points on scale) |
12 (5) |
0 |
0 |
Table 2
Comparison of the quality of completed in-training assessment reports across the three versions of the forms
|
Completed Clinical Evaluation Report Rating (CCERR) rating scale [11] |
Maximum possible score |
Baseline |
Intervention 1 |
Intervention 2 |
F‑value MANOVA |
p-value MANOVA |
|---|---|---|---|---|---|---|
|
Average score (standard deviation) | ||||||
|
Checklist/numeric ratings show sufficient variability to allow identification of relative strengths and weaknesses of the trainee |
5 |
2.9 (0.9) |
1.2 (0.4) |
1 (0.2) |
357.59 |
<0.001 |
|
Comments are balanced providing both strengths and areas for improvement |
5 |
1.5 (0.7) |
2.4 (0.8) |
2.3 (0.7) |
47.61 |
<0.001 |
|
The trainee’s response to feedback and/or remediation during the rotation is described in the comments |
5 |
1.4 (0.8) |
1.7 (0.9) |
1.5 (0.8) |
3.47 |
0.03 |
|
Comments justify the ratings provided |
5 |
2.3 (0.5) |
2.5 (0.5) |
2.6 (0.4) |
6.97 |
<0.001 |
|
Clearly explained examples of strengths using specific descriptions (not generalizations) are provided in the comments |
5 |
1.1 (0.5) |
1.2 (0.5) |
1.2 (0.5) |
1.45 |
0.24 |
|
Clearly explained examples of weaknesses using specific descriptions (not generalizations) are provided in the comments |
5 |
1 (0.2) |
1.1 (0.3) |
1 (0.2) |
1.62 |
0.20 |
|
Concrete recommendations for the trainee to attain a higher level of performance are provided |
5 |
1.3 (0.6) |
2.3 (0.8) |
2.3 (0.6) |
79.09 |
<0.001 |
|
Comments are provided in a supportive manner |
5 |
3.7 (0.3) |
3.7 (0.4) |
3.7 (0.3) |
1.75 |
0.18 |
|
Overall, this ITAR provides enough detail for an independent reviewer to clearly understand the trainee’s performance on the rotation |
5 |
2.7 (0.7) |
2.9 (0.7) |
3.1 (0.6) |
10.35 |
<0.001 |
|
F‑value ANOVA |
p-value ANOVA | |||||
|
Total score |
45 |
18.0 (2.6) |
18.9 (3.1) |
18.8 (2.6) |
3.33 |
0.04 |
Intervention 1 included splitting the comment box into four specific boxes and moving them to the beginning of the form, and replacing rating scale items to checklist items (pass-fail grading). Intervention 2 included simplifying the comment boxes, reducing the number of checklist items, and providing a hyperlink to a detailed assessment rubric
Statistically significant results are in bold
Fig. 1
Average scores on the nine items of the Completed Clinical Evaluation Report Rating (CCERR) scale [11]. Dash‐lines indicate non‐significant results, continuous lines statistically significant results, with black lines indicating steeper changes
