Comparing the passing rates of nursing students participating in the objective structured clinical examination using fixed score, Angoff, yes/no Angoff, and 3-level Angoff methods†

Assessment, a key component of competency-based medical education, is widely recognized as a valuable method to verify learning achievements. It is not only a tool for verification of learning outcomes but also a driver of motivation, directly influencing students’ attitudes and behaviors toward their learning. The nature of assessment, particularly in medical and nursing education, demands that its results be justifiable to multiple stakeholders, including students, evaluators, institutions, and the wider professional community.¹ The concept of fairness and transparency in assessment is vital to maintaining the integrity of the educational process and ensuring that students’ achievements are accurately reflected.²

In recent decades, there have been significant advances in our understanding and use of different methods of assessment, especially workplace-based assessments.³ These assessments, which closely mirror real-world clinical tasks, have become integral in bridging the gap between theory and practice.⁴ This shift emphasizes the need for assessments that are not only reliable and valid but also applicable to the specific demands of the professional practice that students are preparing to enter.

Nursing students need to acquire not only nursing knowledge but also clinical skills during their studies before graduation. Evidence shows that nursing students must be assessed with different modes of assessment, and the objective structured clinical examination (OSCE) is of utmost importance accordingly.⁴ Among the available assessment methods to evaluate students’ clinical competence, the OSCE is a valid and reliable tool to discriminate between students in terms of mastery over clinical competence. In the OSCE, the elements of competency are assessed in a structured way, with a focus on maintaining objectivity.⁵ Nursing curriculum has also put a considerable emphasis on OSCE implementation for nursing students.⁶

At our University, apart from the formative assessment practices, the OSCE, as a high-stakes examination, continues to be dominant in assessing nursing students’ clinical skills.⁷ The OSCE allows for a systematic and standardized approach to assess clinical skills, ensuring that students are tested on specific competencies through a series of stations, each designed to evaluate different aspects of their clinical performance.⁷ Evidence consistently supports the validity and reliability of OSCEs as a tool for differentiating levels of clinical competence among nursing students.⁸ Moreover, nursing education, which emphasizes both theoretical knowledge and practical, hands-on clinical skills, has increasingly integrated OSCEs as a key component in the curriculum.⁹ The use of OSCEs enables a multifaceted evaluation of nursing students, ranging from technical skills to communication and critical thinking skills.⁷,⁸ However, the growing reliance on this method also raises important considerations regarding its implementation and its capacity to reflect the complexity of nursing practice truly.

In high-stakes examinations, it is highly important to design effective assessments as well as determine the standard setting to differentiate between candidates who have reached the required level of proficiency and those who have not attained the essential professional skills. Standard setting in OSCEs is pivotal for ensuring fairness and consistency in assessing clinical skills, and it defines the minimum competence level required to pass.¹⁰ This process incorporates various methods, among the most commonly used methods are the mean borderline group method, borderline regression, and the Angoff method. Based on the mean borderline group method, examiner opinions provide an overall assessment of examinees’ performance in each station, separate from the checklist score. The average score of all borderline examinees is then used as the passing threshold for that particular station. By calculating the average borderline score for each station, the overall pass mark for the entire OSCE can be determined.¹¹ This approach is seen as a relatively practical and straightforward method but has been critiqued for its subjective nature and potential for inconsistency.¹²,¹³ Additionally, the borderline regression method involves using linear regression to predict total OSCE scores based on global ratings. Pass marks are determined by substituting the scores of borderline candidates into the regression equation for each OSCE.¹⁴ However, critics argue that it may fail to capture the nuanced, multifaceted nature of clinical competence, as it assumes that performance across stations is linear, which may not always be the case.¹⁵ In the modified Angoff method, judges examine each question following the identification of a marginal candidate and determine whether the candidate would answer correctly.¹⁶ This method is considered one of the acceptable methods of setting different pass marks.¹⁷ However, it has been criticized for its time-consuming nature, complex organization, expense, and the significant variation in judgments that can arise among raters.¹⁸ All in all, research findings accentuate that setting consistent standards for evaluating students’ performance systematically and objectively is essential in medical education.^15,19

A recent study by Park²⁰ found that the Angoff method was more reliable than the yes/no Angoff method in estimating the cut score of the Korean Medical Licensing Examination (KMLE). However, the yes/no Angoff method with a 60% probability for deciding “yes” can also be used as a substitute for the Angoff method in estimating the cut score of the KMLE. In a study aimed to compare a rational and an empirical standard-setting procedure conducted by Kramer et al.¹⁸, the borderline regression method provided a more credible and reliable standard for an OSCE than a modified Angoff procedure. The researchers concluded that a reality check improves the credibility of the Angoff procedure but does not improve its reliability. A study by Hejri et al.²¹ assessed the reliability of borderline regression method when the pass-fail standard in an OSCE was calculated by averaging the borderline regression method standards obtained for each station separately and found that the root mean square error of the standard was very small indicating that borderline regression method is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance. A study conducted by Dwivedi et al.²² at Xavier University School of Medicine in 2020 compared 4 different standard-setting methods with OSCEs. This study indicated that the traditional standard-setting method resulted in the highest failure rates. In contrast, the modified Angoff method and borderline regression method demonstrated consistent reliability and practical suitability, with an average of these methods providing an acceptable cutoff score. In a similar vein, Dwyer et al.¹⁷ at a Canadian University in 2016 used the Angoff method to set pass marks. This was then compared to those determined by the Borderline Group and Borderline Regression methods. This study affirmed the modified Angoff method as a suitable approach for setting differentiated pass marks for varying levels of expertise.¹⁷

Based on recent studies on standard setting, research findings indicate that various methods of setting standards can result in varying levels of acceptability.²³ To the best of our knowledge, there is a paucity of research in the field of Nursing focusing on various approaches to standard setting in nursing OSCEs. Therefore, the purpose of this study was to fill this gap by comparing the frequency of passing scores of nursing students participating in OSCE by 4 methods of fixed score, Angoff, yes/no Angoff, and 3-level Angoff.

Findings revealed that the majority of nursing teachers (n = 13) who took part in standard setting were women (76.9%), married (76.90%) and had a master’s degree (69.20%). Furthermore, the majority of the 65 nursing students who took part in the OSCE were male, accounting for 50.80% of the participants. The age range of the students varied from a minimum of 22 years to a maximum of 49 years.

The data presented in Tables 1–6 show the distribution of student scores and the frequency of passing across different stations using various standard-setting methods.

As can be seen from Table 1, in pharmaceutical care, vaccination, gavage, and IV therapy stations, the passing percentage in OSCE using the Angoff score method was significantly higher than the passing percentage using the fixed score method. There were no significant differences observed in other stations.

Furthermore, as depicted in Table 2, the frequency of passing in OSCE using the fixed score method at vaccination, gavage, and IV therapy stations showed a significant difference compared to the frequency of passing obtained using the yes/no Angoff method. In this regard, the fixed score method resulted in a higher acceptance rate at the vaccination and gavage stations, while the yes/no Angoff method yielded a higher acceptance rate at the IV therapy station.

According to the data presented in Table 3, the frequency of passing scores in the OSCE at the gavage and IV therapy stations was significantly higher using the 3-level Angoff method compared to the fixed score method.

As shown in Table 4, in the gavage station, the frequency of achieving a passing score in OSCE through the Angoff method was significantly higher than the passing score obtained through the yes/no Angoff method. Conversely, at the IV therapy station, the frequency of passing scores using the yes/no Angoff method exceeded those achieved through the Angoff method. These distinctions were found to be statistically significant.

Table 5 indicates a notable difference in passing score frequencies between the Angoff method and the 3-level Angoff method for gavage and IV therapy stations. Specifically, the passing score frequency was significantly higher when the 3-level Angoff method was used compared to the standard Angoff method in these stations.

According to Table 6, results at the gavage station showed a higher frequency of pass scores in OSCE when using the 3-level Angoff method compared to the yes/no Angoff method. Conversely, at the IV therapy station, the frequency of pass scores was higher with the yes/no Angoff method than with the 3-level Angoff method, with a statistically significant difference.

This study was conducted to compare the frequency of passing of nursing students participating in OSCE by different standard-setting methods (fixed score, Angoff, yes/no Angoff, and the 3-level Angoff). Different studies have been undertaken to compare different standard settings for the objective structured clinical exam.¹⁰’¹¹’¹⁹

As findings revealed, there was a significantly higher rate of passing scores in OSCE for pharmaceutical care, vaccination, gavage, and IV therapy stations when the Angoff method was used to compare the fixed score method. No significant difference was observed in other stations. This may be attributed to the lower standard passing score achieved through the Angoff method compared to the fixed score method in these specific stations. Additionally, the fixed score method relies on a predetermined standard for the test without considering individual examinee difficulty levels. This fixed standard may lead to varying passing rates among students in different exams, as it does not adjust based on student performance. It should be noted that, in this study, the Angoff score was lower than the fixed score in the station using an ice bag; however, the rate of successful completion was the same.

The study undertaken by Jalili and Mortaz Hejri to determine the quorum score and acceptance rate of pre-internship medical students at Tehran University of Medical Sciences in OSCE using 4 methods of fixed score, Angoff, borderline regression, and Cohen showed that despite the acceptance standard of Angoff’s method in 2 stations of genital examination and suturing, which was lower than the borderline regression method, the frequency of acceptance in these stations was the same.¹⁰

Findings also showed that there is a significant difference in passing rates on OSCE assessments using the fixed score method at vaccination, gavage, and IV therapy stations compared to the yes/no Angoff method. Specifically, the passing rates were higher using the fixed score method at the vaccination and gavage stations, while the yes/no Angoff method resulted in higher passing rates at the IV therapy station. We attribute the results obtained for vaccination and gavage stations to the lower passing score of the fixed score method. Similarly, the same result obtained for the IV therapy station can be due to the lower passing score obtained from the yes/no Angoff method. We assert that judges for the yes/no Angoff method take the difficulty or ease of the items into account and, based on the borderline student performance, allocate a score from 0 to 100. Consequently, it seems that judges found the performance at the IV therapy station to be more challenging compared to the vaccination and gavage stations.

Another finding of the present study is that the rate of achieving a passing score in the nursing OSCE, particularly in the gavage and IV therapy stations, is significantly higher when using the 3-level Angoff method compared to the fixed score method. This finding may be attributed to the lower standard-setting score and the perceived simplicity of the 3-level Angoff method in comparison to the fixed score method.

Another finding from this research indicated a significant increase in acceptance rates during OSCE evaluations at the gavage station when using the Angoff score method as opposed to the yes/no Angoff method. In contrast, a higher frequency of passing scores was observed in the IV therapy station when using the yes/no Angoff method compared to the Angoff method. This difference was found to be statistically significant. This discrepancy may be attributed to the lower acceptance standard score in the Angoff method at the gavage station and the leniency of judges in the yes/no Angoff method at the IV therapy station.

Another finding of this study showed a significant difference in the frequency of passing scores in OSCE at gavage and IV therapy stations using the Angoff method compared to passing scores obtained through the 3-level Angoff method. In the gavage and IV therapy stations, the frequency of passing scores using the 3-level Angoff score method was significantly higher than the passing scores using the Angoff scoring method. One possible explanation for this phenomenon is the lower standard scores obtained by the 3-level Angoff method compared to the Angoff method. It is hypothesized that the judges using the 3-level Angoff method are limited to selecting scores of 0%, 50%, or 100% for borderline students at a station. If they do not choose 100, they must choose either 0% or 50%. However, judges using the Angoff method have the flexibility to choose scores within a range of 0–100. As a result, there is a likelihood that judges using the Angoff method may select scores between 50 and 100, leading to a higher passing score. Kaufman et al.²⁴ believe that Angoff’s method may lead to setting a very high and strict standard.

Our research findings indicate that there was a significantly higher frequency of passing scores in the OSCE at the gavage station when using the 3-level Angoff method compared to the Yes/No Angoff method. Conversely, at the IV therapy station, the frequency of passing scores using the Yes/No Angoff method was higher than those using the 3-level Angoff method, with statistical significance. We suggest that the discrepancy in passing rates between these 2 stations is a result of the lower acceptance standard set by the 3-level Angoff method at the gavage station and a lower acceptance standard score at the IV therapy station.

This study has its own limitations. They include: (1) a lack of experienced panelists who are not familiar with the standard setting for the nursing OSCE. (2) focus on determining the acceptance standards based on test-oriented approaches rather than examinee-oriented methods. (3) the paucity of relevant studies in line with our research. (4) Limited external validity of study results. While the findings on the comparative effectiveness of different standard-setting methods in the OSCE are interesting, they may not apply universally to other nursing programs, institutions, or countries. The sample size is small, and the sample itself may not be fully representative of the broader population of nursing students and educators. Additionally, the specific clinical stations and methods tested may not be applicable to other contexts with different priorities or curricula. Further studies with larger and more diverse samples, conducted across multiple settings, would be necessary to better understand the generalizability of these findings.

Future research can focus on the design and implementation of studies to determine the relationship between the pass/fail scores and their frequency, using different methods such as borderline regression, Cohen, etc.

This study examined 4 different approaches to standard setting: fixed score, Angoff, yes/no Angoff, and 3-level Angoff, to analyze the passing rates associated with each method. Due to the diversity in standard-setting approaches and OSCE stations’ design, it was not possible to find similar studies that investigated the acceptance rates across different stations. Nonetheless, it is evident that despite judges’ consistency in scoring methods, varying approaches in determining acceptance standards and judges’ strictness or leniency can result in disparate scores and acceptance frequencies. These discrepancies, which have been noted in previous studies, can be statistically significant.