Fig. 1
Schematic overview of competency-based assessment program at the Faculty of Veterinary Medicine, Utrecht University. Mini-CEX mini clinical evaluation exercise, MSF multisource feedback, SA self-assessment, EBCR evidence-based case report, PDP personal development plan
Table 1
Assessment methods, domain and item scores included from 1 January 2012 until 6 July 2016
|
Total |
Included (≤T124) | ||
|---|---|---|---|
|
Assessment methods |
Total |
17,991 |
16,575 |
|
Mini-CEX |
8,013 |
7,899 | |
|
MSF |
8,787 |
7,514 | |
|
SA |
1,191 |
1,162 | |
|
Domain scores |
Total |
134,938 |
124,649 |
|
Mini-CEX |
59,362 |
58,595 | |
|
MSF |
67,160 |
57,838 | |
|
SA |
8,416 |
8,216 | |
|
Item scores |
Total |
363,526 |
327,974 |
|
Mini-CEX |
89,416 |
88,267 | |
|
MSF |
241,604 |
207,952 | |
|
SA |
32,506 |
31,755 | |
The total number of assessment data points collected in the program of assessment compared with those analyzed in the study. Difference are due to some students exceeding the maximum length of program (e. g., due to remediation, sickness)
Mini-CEX mini clinical evaluation exercise, MSF multisource feedback, SA self-assessment
Table 2
Generalizability analysis by method
|
Source |
MSF |
Mini-CEX |
SA | |||
|---|---|---|---|---|---|---|
|
σ2 |
% |
σ2 |
% |
σ2 |
% | |
|
Week (w) |
0.02 |
3.95 |
0.06 |
11.49 |
0.03 |
10.85 |
|
Student(s)|week |
0.16 |
43.16 |
0.21 |
41.98 |
0.12 |
45.35 |
|
Competency (c)|week |
0.01 |
2.89 |
0.03 |
5.35 |
0.02 |
6.98 |
|
s|w*c|w, error |
0.19 |
50.00 |
0.21 |
41.19 |
0.10 |
36.82 |
|
Total |
0.38 |
0.51 |
0.26 | |||
|
G-coefficient (Ep2) |
0.86 |
0.88 |
0.90 | |||
σ2- variance component, % percent, G‑coefficient =
Multisource feedback (MSF), mini clinical evaluation exercise (Mini-CEX), self-assessment (SA) for week (n = 124), student nested within week (n = 962 students), and competency nested within week (n = 7 competency domains).
Fig. 2
Development of performance (score) dependent of student. The Y‑axis represents the average score of students’ performance per week on a 5-point Likert-scale. The average score per week is collapsed per competency domain, per method and per student. The X‑axis represents 124 weeks of clinical training. The error bars represent the standard error (SE)
Fig. 3
Average competency domain score (µ, se) within student across competency domain discretized by four weeks. Y‑axis represents the average score of students’ performance per competency domain on a 5-point Likert-scale. The average score is collapsed per method and per student. The error bars represent the standard error (SE)
Table 3
Multilevel random coefficient models, where repeated measures (residual, Level 1) are nested within Competency Domain (Level 2), Assessment Methods (Level 3) and Students (Level 4). Model 2–4 assess the effect of linear and non-linear time (inflection points, cubic and quadratic) on repeated measures
|
Parameter Effects |
Model 1 (SE) |
% of variance |
Model 2 (SE) |
% of variance |
Model 3 (SE) |
% of variance |
Model 4 (SE) |
% of variance |
|---|---|---|---|---|---|---|---|---|
|
Fixed effects | ||||||||
|
Intercept |
3.876 (0.009)* |
3.487 (0.014)* |
3.405 (0.015)* |
3.487 (0.015)* | ||||
|
Week |
0.008 (0.0002)* |
0.0122 (0.003)* |
0.002 (0.001)* | |||||
|
Week * Week |
−3.88 × 10−5 (1.9 × 10−6)* |
0.0002 (1.9 × 10−5)* | ||||||
|
Week * Week * Week |
−1.17 × 10−6 (5.4 × 10−8)* | |||||||
|
Random effects | ||||||||
|
Level 1: Repeated Measures (Residual) |
0.308 (0.001)* |
46.53 |
0.308 (0.001)* |
61.11 |
0.307 (0.001)* |
59.73 |
0.306 (0.001)* |
60.41 |
|
Level 2: Competency Domain (Intercept) |
0.014 (0.001)* |
2.11 |
0.014 (0.001)* |
2.78 |
0.014 (0.001)* |
2.72 |
0.014 (0.001)* |
2.72 |
|
Level 3: Assessment Method (Intercept) |
0.073 (0.003)* |
11.03 |
0.068 (0.003)* |
13.49 |
0.069 (0.003)* |
13.42 |
0.065 (0.003)* |
12.90 |
|
Level 4: Student (Intercept) |
0.267 (0.020)* |
40.33 |
0.114 (0.008)* |
22.62 |
0.124 (0.009)* |
24.12 |
0.122 (0.008)* |
23.98 |
|
Level 4: Student (Covariance) |
−0.005 (0.0003)* |
−0.002 (0.0001)* |
−0.002 (0.0001)* |
−0.002 (0.001)* | ||||
|
Level 4: Student (Slope) |
9.53 × 10−5 (5.0 × 10−6)* |
0 |
2.61 × 10−5 (1.7 × 10−6)* |
0 |
2.76 × 10−5 (1.8 × 10−6)* |
0 |
2.57 × 10−5 (1.7 × 10−6)* |
0 |
|
−2 Loglikelihood |
219,200.57 |
218,308.86 |
217,907.54 |
217,488.23 | ||||
|
Degrees of Freedom (DF) |
7 |
8 |
9 |
10 | ||||
SE standard error, % percent
*P < 0.001