Figure 1
Inherent tensions to validity in the era of competency-based health professions education.
Table 1
Paradigms of assessment and application to validity in competency-based health professions education.
| PARADIGMATIC ORIENTATION | DEFINITION | TYPES OF DATA FAVORED | EXAMPLES OF EVIDENCE | USE IN COMPETENCY-BASED HEALTH PROFESSIONS EDUCATION |
|---|---|---|---|---|
| Post-positivism | There is a “true score” that reflects a learner’s ability in a specific competency domain | Objective, numerical, standardized data | Classical test theory Item-response theory Inter-rater reliability | Standardized test settings (e.g. multiple-choice item tests, objective structured clinical examinations, etc.) |
| Constructivism | There is no “true score,” the definition of competence is socially constructed | Subjective, qualitative, non-standardized data | Thematic analysis Generalizability theory* | Workplace-based assessments |
| Pragmatism | Competence is developed through a mutual understanding and “workability” between groups | Supports a combination of qualitative and quantitative data | Effective group process | Group decision-making (i.e. competency committees) |
[i] *Generalizability theory can offer sources of variance (i.e. raters vs. learners) but cannot provide a more specific delineation within the source (i.e. meaningful rater differences vs. construct-irrelevant variance).