How Learning Culture Influences the Survivability of an Online Feedback Tool

Setting and innovation

The study took place at the ED of a tertiary paediatric hospital which attends to 170,000 patients a year and where 40 to 50 trainees rotate through every six months for training. SA and AT developed an online written feedback tool (refer to Appendix 2) to improve the quality and quantity of written feedback provided by faculty to trainees in the ED. “Faculty” refers to the senior doctors in the ED who play the roles of clinicians, educators and supervisors. Each senior doctor has between 10 to 30 years of supervisory experience.

In our ED setting, the online written feedback tool was made accessible to faculty who served as either “clinical supervisor” or “shift supervisor”. The term “clinical supervisors” refers to faculty who are assigned to be the overall assessor and supervisor of a trainee throughout the posting and may not necessarily work with the trainee during every shift due to the nature of shifts in the ED. Clinical supervisors were meant to be the recipients of the written feedback so that they can provide specific and actionable guidance to their trainees for improvement. The term “shift supervisors” refers to faculty who work with trainees during shiftwork. As shift supervisors had opportunities to observe trainees’ interactions with patients and other healthcare professionals, their written feedback was thought to hold tremendous value for improvement.

The online written feedback tool was developed to address a significant problem: the trainees’ assigned clinical supervisors did not always have sufficient contact with the residents throughout their posting to have a holistic view of their performance because of shift work. This meant that at scheduled meetings, clinical supervisors often could not provide sufficient feedback based on their personal observations to help the supervisees improve. SA and AT thus designed the tool to enable the longitudinal collection of written feedback from multiple faculty members who worked with trainees in the ED. The tool was designed to encourage faculty to provide feedback across all 6 of the Accreditation Council for Graduate Medical Education-International (ACGME-I) core competencies. The tool could be conveniently accessed by faculty via their smart phones to submit real time written feedback related to clinical consultations, procedural-skill observations, simulation sessions, didactic teaching, presentations or research-related work. The collection of written feedback from various faculty was intended to provide a more holistic picture of trainees’ performance based on observations. Faculty were also taught how to use the R2C2 framework (R-Relationship, R-Reactions, C- Content, C-Coaching) [20] during supervisor-supervisee meetings. R2C2 allowed faculty to have more meaningful discussions and reflections during those scheduled meetings with their trainees during their posting. As teaching faculty with prominent roles in medical education, SA and AT collaborated with faculty to innovate feedback practices to improve the quantity and quality of feedback for trainees.

Online Feedback Tool: Development and Learnings Guiding its Evolution

This segment reflects the critical learnings in the journey of developing and implementing the tool. The design and implementation of our online written feedback tool were guided by self-determination theory (SDT) [21] and diffusion of innovation theory (DIT) [22]. SDT explains motivation by emphasizing an individual’s inner resources, fostering sustained, self-regulated behaviours [23] and informing faculty development initiatives [24, 25]. DIT is a framework that explains how new ideas, technologies and practices spread through a social system over time and plays a vital role in understanding change in healthcare, education, organisational and implementation contexts [22]. However, despite careful considerations in the design, delivery, and implementation, as well as multi-pronged faculty interventions (illustrated in Diagram 1), the tool faced significant challenges. Despite receiving strong leadership support from the head of the department, there were variable levels of adoption of the tool among faculty. There were difficulties in motivating faculty to continue providing feedback via the tool. Although we collected and disseminated positive resident feedback highlighting the quality of faculty-resident interactions facilitated by the tool and residents’ appreciation for faculty engagement with the feedback process, achieving sustained and consistent faculty adoption of the tool proved difficult. Faculty were also overwhelmed with clinical duties and felt that the tool was a duplication, if not an additional burden, to the existing forms they already had to complete as clinical supervisors. Despite faculty development and regular meetings to engage the faculty, there was only transient improvement in their motivation and provision of feedback for trainees among faculty. The challenges of adoption and sustained use of our online feedback tool highlighted the importance of designing tools with survivability in mind, so that the tools can actually withstand institutional changes over time [14]. This tool was launched and used for nearly two years and about 80% of our faculty utilised the tool during this period. Over time, the percentage of faculty using the tool and the number of feedback submissions by faculty declined. The initial aspiration (yellow box in Diagram 1) did not materialise despite all efforts and interventions. The team was committed to re-examine the factors affecting the survivability of the feedback tool and the challenges to achieving our aspiration. We aimed to look beyond individual motivation to larger contextual perspectives within the learning culture that would inform the tool’s long-term survivability.

Diagram 1

Recruitment and participants

Twenty-two clinical faculty from the pediatric ED at KK Women’s and Children’s Hospital (KKH), Singapore, volunteered to participate following an email invitation from the department’s administrative team. Ten faculty were interviewed based on their background and patterns of use of the online feedback tool, These faculty members worked with trainees in various roles and demonstrated varying levels of tool adoption and usage – as posting supervisors during ED rotations and/or as clinical supervisors during routine clinical shifts. The sequence of interviewing was as follows: the initial three participants were ‘frequent users’; the subsequent two participants were infrequent users who became non-users; the next two participants were initial adopters who became non-users; and the last three participants were a combination of initial non-users or infrequent users who became late adopters of the tool. We adopted theoretical sampling, a key feature in CGT, whereby participants were selected strategically to further explore propositions and gather contrasting perspectives [16, 29]. Table 1 shows the definitions of the terms used in defining the frequency and length of the tool’s usage among faculty. Table 2 illustrates the profiles of the participants and their involvement in the various phases of the interviews.

Data Collection

One-on-one semi-structured interviews, guided by a topic guide (Appendix 1), were conducted via Zoom with the participants and lasted 45–60 minutes. Written informed consent was obtained. There were two interviewers, SA and AT. The interviewers strategically planned and arranged themselves to interview specific participants to improve rapport-building, build trust and minimize power differentials among participants to elicit in-depth narratives. This approach has been extremely valuable in qualitative studies [17]. The audio-recorded interviews were transcribed by an independent party to ensure anonymity, accuracy, and standardization for ease of coding and analysis. Data collection occurred simultaneously with data coding. The experiences, incidents, and perspectives of our participants helped identify concepts that shaped the next selection of participants for the interviews. The first three participants (as frequent users of the tool) provided insights into why they were motivated to use the tool longitudinally. The next group of participants, initial adopters who then became non-users, helped us explore the reasons for ‘drop-out’ and disengagement with the tool despite their initial confidence in and adoption of the tool. The final group of participants, initial non-users who later decided to adopt the tool, helped us explore why the tool did not initially appeal to them despite their being strong believers of feedback, and what subsequently changed their beliefs and decisions to adopt the tool. All these strategies fit the iterative nature of CGT to explore discrepancies as well as deepen propositions [16, 30].

Analysis

The transcripts of the interviews were analysed independently by SA, AT and YY, using manual line-by-line coding as well as NVivo software (QSR International Pty Ltd, Melbourne, Australia, 2018). A combined manual-software analysis provided opportunities for researchers to explore their data visually and strategically to foster creative thinking and stimulate the construction of insights that enhance the analytic process [31]. Researchers wrote memos to enhance reflexivity [19] and held regular meetings for constant comparison of their analysis and tweaking of the interview guide which informed the choice of subsequent sampling. LL was consulted regularly throughout the study and provided guidance on the development of the research question, expansion and deep exploration of propositions, as well as refining the interview guide to support theoretical sampling.

After 10 interviews, we determined that we had reached theoretical sufficiency in terms of being able to robustly describe how learning culture influenced the survivability of our online written feedback tool based on its perceived usefulness [32, 33].

Ethics

This study (EING1909) was approved by the SingHealth Centralised Institutional Review Board (CIRB number 2024/2049).

The lack of support systems exacerbated the tension among faculty as clinical teachers

In healthcare education, faculty were often expected to adopt dual roles as clinicians and teachers. Yet, the LC of the clinical learning environment privileged service delivery, sidelining faculty’s need to be better clinical teachers. This invariably left them feeling that “a lot of feedback and teaching was by the way” and became “mostly accidental rather than systematic.” (P1)

Participants were fully aware of the importance and utility of the feedback tool. However, the heavy clinical workload constrained the full utilisation of the tool and this was one of the factors affecting its survivability:

This tension of having to balance patient care yet shouldering the responsibility of teaching and guiding new trainees was not new. However, with the lack of support systems and further beliefs illustrated below, it became a lingering shadow that influenced the tool’s survivability.

Misapprehensions about written feedback manifest as misgivings towards the tool

Faculty’s incomplete understanding of the nature and purpose of written feedback (WF) had direct implications on the tool usage. Multiple misapprehensions emerged, each contributing to the perception that written documentation of feedback was unnecessary, duplicative, or reserved only for serious situations rather than serving its intended role in longitudinal trainee development.

Many participants expressed the view that written feedback was redundant when they had already provided verbal feedback. Faculty questioned the utility of documenting feedback that trainees had already received verbally, assuming that the spoken word was sufficient for learning:

The perception of redundancy extended to beliefs about the purpose and audience of written feedback. Some faculty felt that written feedback served administrative rather than developmental purposes, and assumed that trainees would remember verbal feedback without written reinforcement. Hence, faculty could not appreciate the value of feedback being documented again in the tool:

The misapprehension also manifested in selective documentation practices based on perceived severity thresholds. Some faculty believed that only serious or extreme feedback needed to be documented, overlooking the value of capturing routine formative observations or encounters for development tracking. This threshold mentality paradoxically created hesitancy around negative feedback due to its perceived permanence and weight and thus, influenced their decision to utilize the tool in documenting their feedback:

Underlying these misapprehensions was a strong belief in the superiority of verbal feedback for learning. Faculty valued verbal feedback for its ‘timeliness’ and ‘effectiveness’ and contextual immediacy, not recognizing that WF could serve a complementary longitudinal function rather than a competing one:

Faculty often perceived WF as redundant since they had already provided verbal feedback. They failed to recognize that both forms of feedback served distinct and complementary purposes in a feedback system. This fundamental misunderstanding shaped their engagement with the tool, as many believed verbal feedback alone was sufficient for trainee development.

Low psychological safety undermined trust in the tool’s intent

The department’s hierarchical and judgmental LC led to low psychological safety. Bedside tutorials, didactic teachings as well as morbidity and mortality rounds were often frowned upon as ‘clown shows’, whereby feedback delivery was perceived as a form of humiliation:

When the LC was marked by low psychological safety, faculty interpreted feedback processes, especially those involving documentation, as potentially risky or threatening. This perception of threat undermined trust in the tool itself, reducing its acceptance, adoption, and survivability.

The healthcare system stored documentation as a permanent record. Thus, feedback captured in writing was also deemed to be a permanent ‘trace’ in the records that could adversely influence the trajectory of a trainee’s development:

Participants shared the lack of trust with regard to who actually had access to WF. The tool became an extension of the LC that surrounded it. In an environment lacking safety, trust in the tool’s neutrality was undermined. Multiple stakeholders in the existing system were perceived to always have access to the feedback:

Faculty shared how they felt feedback recorded in the tool then actually “passed through layers of interpretation” (P3) by supervisors and the administrative team, risking distortion of the original intent. Faculty found it hard to fully trust the tool to deliver its intended purpose of developmental feedback within the learning environment. Even when reassurances were given about the developmental intent of the feedback tool, the fear of misinterpretation, emotional exposure, or reputational consequences lingered:

In a LC with low psychological safety, the lack of trust inadvertently extended to the tool mediating feedback exchanges. A feedback tool that carried documented feedback was deemed to risk permanence, misinterpretation, or escalation. Despite repeated assurances, faculty appraised the feedback tool as a conduit through which judgment could flow, rather than as a developmental aid. This appraisal, rooted in fear, constrained engagement and inhibited the tool’s survivability.

Strong feedback loops were essential for tools to survive

As with many healthcare settings, the LC was made up of opportunistic, episodic encounters between faculty and trainee. From an assessment and feedback perspective, this was suboptimal as opportunities for longitudinal observation and assessment from one supervisor would not be possible given the nature of rotations and shift work. The fleeting nature of these relationships weakened faculty’s ability to close the feedback loop, where they could see whether their feedback had been accepted and used by trainees. Given the circumstances, participants described feedback as “like a kind of recovery or repair process” that helped the trainees improve, yet noted that “we never know [the] long term [impact] because many times it really doesn’t come back to us.” (P2)

Without strong feedback loops that demonstrated how faculty’s written observations contributed to trainee development, many struggled to justify the effort required to use the tool consistently. Faculty expressed frustration at not knowing if their efforts made any difference and what the ‘downstream impact’ was:

The feedback loops were also crucial to many faculty as they were seen as an important channel to receive feedback from trainees for their own reflection and improvement. The opportunities for feedback on feedback were unfortunately not encountered frequently in the current feedback system:

The lack of closure meant that faculty invested time and effort into using the tool to provide WF without any confirmation that it was read, understood or applied by trainees. Patient-care responsibilities rendered time-starved faculty further inability to develop meaningful relationships with trainees they met opportunistically in the ED. The episodic nature of encounters made faculty question the appropriateness of their efforts:

Beyond knowing whether feedback led to change, faculty needed signals that their feedback was valued and sought after by trainees. When trainees actively sought feedback, it was perceived that they valued it and faculty would be more motivated to use the tool:

This distinction was critical – participants valued learner-initiated feedback requests, rather than program mandates to provide feedback in a paternalistic manner. Trainees actively seeking feedback created a feedback loop that reinforced faculty effort by demonstrating immediate appreciation and purpose, reminding faculty that the time and effort to give feedback to a trainee was crucial as “for us [as a faculty or supervisor], it [this reminder to give feedback] becomes immediately to my consciousness.” (P4).

Thus, strong feedback loops associated with the tool actually fed faculty with enough information to consistently remind them of the tangibility of their efforts in providing feedback to trainees, which was essential to ensuring the tool’s survivability.