Table 1
Academic titles of focus group participants.
| Academic Assessment Lead | Academic Lead Assessment | Acting Dean |
| Associate dean | Associate Dean, Learning and Teaching | Associate professor (3 participants) |
| Chief Examiner and Head of Assessment | Director of Assessment | Director Medical School |
| Discipline Leader | Doctor of Medicine Program Director | Faculty Dean |
| Head of Assessment |
Figure 1
The components of fair judgement.
Table 2
Fair judgement demonstrated as a complex adaptive system.
| FEATURES OF COMPLEX ADAPTIVE SYSTEMS | AN EXAMPLE OF HOW THIS RELATES TO FAIR JUDGEMENT |
|---|---|
| Medical schools need to determine if students meet the standard expected to graduate. | |
| COMPLEX CAS consist of individual agents [25] who make independent choices about their actions [29]. Each individual agent reacts to what the other agents are doing [2830]. This interaction between the agents directs the CAS and influences the outcomes it produces [2728]. The principle of connectivity is that a system’s behaviour relies less on the nature of the individual agents than on the quantity and quality of connections between them. Therefore learning how things are interconnected is often more useful than learning about the pieces [29]. Despite the unpredictable and adapting nature of complex systems, principles and patterns arise [28]. Understanding these patterns is fundamental to understanding how the system works [26] as they guide behaviours within it [28]. | Judgement decisions are made by a diverse group of individuals or committees considering multiple different assessments and evidence. Assessors are independent experts allowing them to make independent judgement decisions depending on their interaction with the data and other individuals. It is not possible to create specific rules for how judgement decisions are made. Each judgement decisions will involve different data, with different circumstances and will be perceived in different ways. Furthermore, the determination of the outcome is more than simply including more measurement points in the model. Although further data may improve judgement decisions, the interactions between these factors also needs to be considered. Expert assessors recognise that a multitude of factors should be considered in assessment and can perceive information from multiple interactions simultaneously process this information to identify patterns. Making meaning of these relationships is encouraged. |
| ADAPTIVE The efficacy and effectiveness of CAS is mainly due to the adaptability of the system. Agents adapt to past experience [2931], internal and external influences. However this also leads to unpredictability [262832], and resistance to centralised control [33]. Control is dispersed; the result of a huge number of decisions made by individual agents [31]. Work arounds and muddling through are central to CAS [2932]. Tensions and paradox do not necessarily need to be resolved [25]. Order, innovation and progress emerge naturally from the system, they do not need to be imposed from within or from outside [3234]. Seemingly obvious interventions can have minimal impact on system behaviour, whereas small changes can have large unintended consequences [283031]. | The assessors and the system of assessment are adaptive. Previous experience, new information, a different assessment method or a change in expectations causes the agents and thus the system to change. Adaption is often enhanced in crisis, this may be seen in the case of a struggling trainee, making decisions with incomplete data or changing environments such as pandemics. Agents self-organise to consciously improve the interactions between patients, learners, the environment and the university to ensure judgements are fair. The desire is often to apply more rules, however these rules alone are less likely to influence judgement decisions. If a judgement is not obvious, the system is still able to move forward and judgement decisions made. Effective judgements can emerge, even from minimum initial data. There will always be tensions when making judgement decisions. For example between what is fair for the patient and what is fair for the individual student, or balancing learning with assessment. |
| SYSTEMS Complexity thinking maintains that systems can be aided by a minimal structure, such as fuzzy, ill-defined boundaries [29]. These boundaries act as constraints in that they provide a stable structure within which change can occur [2632]. Individual agents and CAS are embedded within wider CAS. Therefore, we cannot fully understand the individual agents or systems without reference to the others [2533]. | Within assessment, boundaries, ground rules and processes, can provide assessors with security and confidence to make judgement decisions. To ensure fair judgement, sufficient organisational structure is needed to keep stakeholders focused on the task, without limiting flexibility, initiative and commitment to overall improvement. Humans are not limited to one identity, but are also members of clinical workplaces, families and social groups which are embedded within cultural environments and wider society. These external memberships influence how agents behave and the perspectives they bring to judgement decisions. |