Fig. 1
CIPP evaluation model. OSCE objective structured clinical examination, TBL team-based learning, UME undergraduate medical education
Table 1
Context evaluation: methods and results
|
Method |
Details of the method |
Main results |
|---|---|---|
|
1. Students, faculty and administrator focus groups |
To explore the challenges of the traditional programme, 21 focus group sessions (12 with students, 7 with faculty and 2 with administrators) were conducted during May to July 2006. Each session were lasted 120–150 min. Data were analyzed using qualitative content analysis method |
Four categories of challenges have been identified: 1. Challenges of the structure of the programme 2. Challenges of the content of the curriculum 3. Limitations of the resources 4. Challenges of the programme implementation |
|
2. Graduation survey |
To evaluate the perceptions of our graduates regarding the quality of the traditional programme, a 262-item questionnaire was developed based on the graduation survey by the Association of American Medical Colleges. A total of 183 questionnaires were completed by medical students upon their graduation from the medical school in 2007 |
- Satisfied with the medical training they received (28.4 %) - Basic science courses lacked clinical relevance (77 %) - Acquired adequate knowledge and skills to start residency training (33.3 %) - Have not been taught sufficient clinical skills in preparations for their future practice (70 %) |
|
3. DREEM questionnaire |
To evaluate the educational environments from perspectives of the students, a total of 541 students (103 basic sciences, 103 preclinical and 335 clinical students) completed the standard DREEM questionnaire in 2008 |
Overall DREEM questionnaire score was 91.46/200 (students’ perception of teaching [23.75/48], students’ perception of teachers [19.42/44], students’ academic self-perceptions [13.21/32], students’ perceptions of atmosphere [23.35/48], students’ social self-perceptions [13.99/28]) |
|
4. A self-study of programme in comparison with national undergraduate medical education standards |
A self-study of the traditional programme was conducted on the basis of the national standards (including 9 domains and 57 standards) in 2007. 234 questionnaires were completed by students, faculty and administrators. 82 department deans, course directors and faculty used the results to evaluate the programme quality in comparison with each national standard on a scale from 0 to 100 |
Final results showed that 22 (40 %) standards were rated as ‘relatively match’ (50–75) and ‘completely match’ (75–100) by more than 50 % of the members of the workshops. 32 (55 %) standards were rated as ‘does not match’ (0–25) and ‘slightly match’ (25–50) by more than 50 % of the members of the workshops |
|
5. Results of OSCE |
86 students participated in an OSCE exam at the end of the clerkship period. The exam was conducted in the morning (2 different tracts) and in the evening (2 similar tracts). Each tract consisted of seven stations |
There was a significant difference (p < 0.001) between history taking (accounted for the highest points) and procedure (accounted for the lowest points) stations |
Table 2
Process evaluation: methods and results
|
Method |
Details of the method |
Main results |
|---|---|---|
|
1. Student evaluation questionnaire |
An online, 40-item questionnaire on a 5-point Likert scale (ranging from 1 = strongly disagree to 5 = strongly agree) was completed by students at the end of each interdisciplinary block regarding the quality of the blocks. A total of 1004 questionnaires were completed for 10 blocks. Mean response rate for each block was 63 % |
Most students agreed or strongly agreed that: - Block material was appropriate (77.7 %) - Block content was related to and consistent with each other (77.4 %) - Integrated content contributed significantly to their learning (73.6 %) - Interdisciplinary questions were suitable (72.4 %) More students disagreed or strongly disagreed that: - Enough time was allotted for subjects (56.9 %) - Block instructors used techniques like questioning to make sessions interactive (47.7 %) |
|
2. Student and faculty focus groups |
To identify strengths and shortcomings of the implemented revised programme, 15 focus group sessions were conducted during December 2011 to December 2014 (12 sessions with students and three sessions with basic science faculty). Each session lasted 30–90 min which were audiotaped and transcribed |
Strengths: - Integration of basic science subjects - Case-based discussion sessions Shortcomings: - Insufficient coordination among the block teachers - Low quality of some team-based learning sessions - Low quality of some exam questions - Disturbing the faculty schedule - Uncertainty about the success of the programme (at the early stage) |
|
3. Individuals interviews with administrator Review the programme documents |
To identify the extent to which the revised programme was implemented as planned, interviews were conducted with six reform committee chairs. Course syllabi and exam questions were reviewed as well |
Holding lectures and practical sessions as planned - E-learning session considerably less than planned - Gradual decline of the exams quality |
Table 3
Product evaluation: methods and results
|
Method |
Details of the method |
Main results |
|---|---|---|
|
1. Student evaluation questionnaire |
To evaluate the students’ perceptions regarding the quality of the basic science phase and its application to the next phase, a 96-item questionnaire was developed based on the graduation survey by the Association of American Medical Colleges. A total of 136 students (response rate, 51 %) completed the questionnaire four month after completing the basic science phase in June 2014 |
More students agreed or strongly agreed that: - Overall, I am satisfied with the quality of my basic science (53.7 %) More students disagreed or strongly disagreed that: - Basic science content had sufficient illustrations of clinical relevance (50.7 %) |
|
2. Student self-assessment regarding vertically integrated themes outcomes |
23 items of the above-mentioned questionnaire were related to outcomes of the vertically integrated themes |
More students agreed or strongly agreed that: - In practical classes, my behaviour and appearance are appropriate to the medical profession (78.7 %) - I do not hesitate to share my knowledge and ability to my classmate during the group work (77.2 %) |
|
3. DREEM questionnaire |
A total of 102 students (response rate, 44 %) enrolled in 2011 and 197 students (response rate, 87 %) enrolled in 2010 completed the standard DREEM questionnaire after completing their basic science phase |
No significant differences were found between traditional and revised programme in Overall DREEM questionnaire scores. Students in revised programme evaluated the educational environments in 10 items significantly better than students in traditional programme (items 2, 5, 9, 11, 16, 28, 30, 37, 39 and 44) |
|
4. Individuals interviews with faculty |
Individual interviews were conducted with 14 basic science faculty 3 years after running the revised programme. Each session lasted 20–45 min |
Faculty concerns were: - Decreased faculty authority - Disturbed faculty schedule - Gradual weakening of the programme - Influence on student learning - Insufficiency of team-based learning |
|
5. Results of achievement tests |
To compare the student performance in exams in the revised and traditional programme, a total of 724 exams results were extracted related to the: - 231 students of the revised programme enrolled in 2011 - 225, 184 and 184 students of the traditional programme enrolled in 2010, 2009 and 2008, respectively |
No significant differences (p s > 0.05) were detected between traditional and new programme in: Grade point average Failure rates National Comprehensive Basic Science Exam result |