Recommendations from a Fellowship Programme for Advocates of Good Software Practices in Research

Introduction

The UK Software Sustainability Institute (SSI) was founded in 2010 to promote the use of good software practices in research and support those developing software in academia, with the overarching aim of improving the quality and impact of research findings. As part of this mission, the SSI Fellowship programme was [1], recruiting around 18 fellows each year and supporting them to develop their knowledge and use of good software practices and to be champions for good practice in their own workplaces and/or domains. The fellowship is undertaken in addition to the fellow’s institutional role rather than being a substantive contract. It consists of an inaugural period of 15 months where fellows are awarded a £3000 stipend and are supported by SSI staff to deliver the advocacy activities they proposed as part of their application.

The programme recruits research professionals who work in roles related to research software at UK research institutes, with a small number of places recently made available for international applicants. As such, fellows are often in relatively newly established ‘non-traditional’ research roles, such as research software engineering and data stewardship, and are unlikely to benefit from the well-defined career pathways available to academics pursuing the ‘traditional’ route from PhD student to professor. They gain access to the multiple cohort SSI fellows’ community and are expected to attend an SSI Collaborations Workshop, the SSI’s flagship event, as well as being encouraged to find a mentor. After this inaugural period, the status of SSI fellow, membership of the fellows’ community, and support from the SSI including access to funds continues with no pre-determined end point. Applications are accepted from professionals at any career stage from PhD candidates or junior Research Software Engineers (RSEs) to senior roles with the largest number being from the mid-career stage cohort.

In this research we sought first to evaluate the benefits of the Fellowship for the fellows and for their institutes and/or research domains, and second, to identify the mechanisms through which these benefits were realised, along with areas for improvement. Our methods primarily focussed on level four of Kirkpatrick’s widely used four-level evaluation model [2, 3] which assesses impact. Additionally, identifying supportive mechanisms partially overlaps with level one, which concerns participant satisfaction with programme quality typically achieved through post-session evaluation surveys. However, our approach drew on the fellows self-reporting the mechanisms that directly related to benefit. Levels two and three of Kirkpatrick’s model, which assess skill and knowledge acquisition and changes in work practices, were beyond the scope of this study. Based on our findings, we present a model of the Fellowship and recommendations for other Fellowship programmes or career development schemes.

Methods

We performed a mixed methods evaluation initially developing and administering a survey to quantify the impact of the Fellowship. We then performed qualitative interviews with a subset of the survey respondents to get a deeper understanding of the benefits, the mechanisms within the Fellowship that gave these benefits, and any areas for improvement. The research was approved by the University of Manchester Proportionate Research Ethics Committee, reference number 2022-15103-25448. Participants gave informed consent for the survey and the interview separately. Pseudonyms have been used for the interview participants when presenting the results in this article. We have made available the survey and interview guide in the detailed report [4] and the anonymised survey data on Figshare [5].

Our first step was to build a model of the Fellowship programme using Theory of Change [6], see Figure 1. This required us to describe the intended impacts for the fellows and their institutes and/or domains, followed by the outcomes for the fellows that would lead to these impacts, and finally, the activities involved in the Fellowship that would produce these outcomes. We used information from the SSI website [1], conversations with the programme team, and findings from a previous Fellowship programme impact study [7] to build this model. It was then refined in workshops with SSI staff, including the programme team and five SSI fellows who were not eligible to participate in the evaluation as they were in a prior cohort. To build the survey, we created items that questioned whether fellows had experienced the outcomes (such as improved knowledge of software good practice) and impacts (such as career development) in the model with open questions to capture any aspects missing from the model. These items were then refined collaboratively with SSI staff. Finally, the survey was piloted with three of the five SSI fellows who were involved in the model development, and their feedback was incorporated into the final version, see Figure 1.

Figure 1

The survey [4] contained seven sections which were: 1) Your fellowship and your role; 2) Skills and network development during your fellowship; 3) Professional development during your fellowship; 4) Impact for your institute(s) and/or research domain(s) and for the individuals that you work with; 5) Support from the SSI during your fellowship; 6) Noticing the benefits of your fellowship; 7) Your final thoughts on the Fellowship program. Most survey items used a 5-point Likert scale, for example ‘strongly disagree’ to ‘strongly agree’, offering a ‘not applicable’ to reduce respondents being forced to select inaccurate responses. Some open text questions were also included to allow respondents to give information not included in multiple choice options or to give further details to their responses. In the final section, survey respondents could choose to receive an email invite to a semi-structured interview. The survey was distributed using the Qualtrics online survey service and was advertised using the fellows’ online forum (Slack) and email list.

The interviews were carried out over Zoom (online video call service). The interview schedule [4] was based on the topic areas within the survey probing the reasons behind participants’ responses. We employed Social and Cognitive Career Theory (SCCT) [8] as a framework to analyse the data as it allowed us to categorise the participants professional growth as advocates of good software practices. SCCT has three strongly inter-related themes: 1) Self-efficacy – a person’s beliefs about their capabilities in a particular context; 2) Outcome expectations – a person’s expectations of successful outcomes from actions or behaviours; and 3) Professional goals, either to engage in an activity or reach a particular attainment level. These three themes were identified in the transcriptions and the data within each of these themes were then coded using standard techniques [9] to give common codes across all six interviews. Finally, quotes were identified that demonstrate the Fellowship develops effective advocates and/or that detail the mechanisms by which advocacy is supported. Areas for improvement for the participants were collated outside of the SCCT analytic framework.

Finally, it should be noted that this research was conducted ‘in-house’ with three of the authors employed by the SSI, with the potential for this conflict of interest to bias the results. In particular, Shoaib Sufi was the SSI Fellowship programme lead at the time of this work. Although he contributed to the conception and design stages, he was not involved in the data collection or analysis in an attempt to limit any bias in the findings.

Participants

All fellows from the 2017, 2018, 2019 and 2020 cohorts were eligible to take part as they had completed their inaugural period. Earlier cohorts were not eligible as they had taken part in a prior impact evaluation survey [7]. 17 complete responses were received for the survey from 71 eligible fellows, giving an apparent response rate of 24%. However, some fellows may not have been in the email list or online forum (Slack), or their email addresses may have been extant, so it is possible that the true response rate may be higher. It should be noted that the 2019 and 2020 cohorts were impacted by the early stages of the COVID-19 pandemic. To compensate for the difficulties experienced during this time, the SSI adjusted the standard 15-month inaugural period to 24 months for the 2019 fellows and 39 months for the 2020 fellows.

Seven of the survey respondents were men, nine were women and one preferred not to give their gender. Two survey respondents came from the 2017 cohort, five from 2018, three from 2019 and seven from 2020. In their respective inaugural year, five respondents considered themselves at a junior career stage (e.g. PhD candidate or junior RSE); nine at an early career stage (e.g. research associate, lecturer, or RSE); and three in mid-career (e.g. senior lecturer or senior RSE). The respondents worked across nine different academic areas including humanities, computer science, biological sciences, and geosciences.

Eight of the 17 survey respondents chose to receive an invitation to an interview with six respondents being interviewed. This met the maximum number of interviews planned. Four of the participants were women and two were men. One was from the 2017 cohort, two from 2018, one from 2019 and one from 2020.

Findings from the survey: areas of strength

We present 3 findings from the survey data. Firstly, most respondents agreed that the Fellowship enhanced their knowledge and their use of good software practices (see Figure 2). Secondly, the Fellowship supported most respondents’ professional development as shown in the six survey items in Figure 3. A respondent commented that “it’s been a confidence boost that I have the potential to be a leader, or to bring about culture change” whilst another explained that “the Fellowship was a great launchpad to many things afterwards that maybe had more direct influence on my career but probably wouldn’t have happened without the Fellowship.” It is interesting to note that there was one outlier respondent who strongly disagreed with all six professional development items, although they gave positive scores in other areas.

Figure 2

Figure 3

Additionally, several respondents (6/14) selected the “probably yes” response to the status of SSI Fellow helping to reduce barriers due to personal characteristics (the three respondents who selected “not applicable” were discounted). Gender, disability, career stage, and being a first-generation graduate were mentioned in the corresponding open text box. One respondent commented that the fellowship “gives me visibility despite my early career stage” and another said that “as someone who suffers great imposter syndrome due to first generation university status and disability status, having the SSI Fellow title gave me the confidence to speak up about RSE best practises.”

The final survey finding showed that all the respondents aimed to deliver activities that benefited their institutes and/or domains, but not always both (see Figure 4), and the majority agreed that they had achieved their aims (see Figure 5). However, there were fewer “definitely yes” responses than for the professional development items. Additionally, whilst the majority agreed that they had introduced others to the work of the SSI, there was less confidence that participating in the programme was essential to delivering their activities (see Figure 5).

Figure 4

Figure 5

There are several threats to validity for the survey findings. First, only 17 out of 71 eligible fellows (24%) completed the survey, which may lead to a non-response bias and limit the degree to which the findings represent to the full fellows’ population. The low numbers also may introduce a selection bias as respondents could disproportionately include fellows who remained engaged with the SSI due to positive experiences. Second, acquiescence bias, due to the dependent nature of the fellows on the SSI, may have resulted in respondents agreeing with favourable statements without the needed critical evaluation. Third, social desirability bias, driven by a wish to appear appreciative and aligned with SSI values, may have led to respondents over-stating benefits and underreporting challenges. Whilst standard steps were taken to reduce these issues, such as guaranteeing anonymity and encouraging responses from fellows that had negative and mixed experiences when advertising the survey, these biases may still have inflated positive perceptions and reduced the reliability of the findings.

Findings from the interviews: areas of strength

Two main findings have been drawn from analysing the interview data using the three SCCT themes of self-efficacy, outcome expectations, and professional goals. First, these data demonstrate that an increase in self-efficacy through engaging with the Fellowship leads to an improved ability to advocate for good software practices. Second, they demonstrate that the Fellowship leads to positive outcome expectations of engaging with the SSI community and an alignment of the participants’ professional goals with their advocacy work.

Interview finding 1: The fellowship increased the participants’ self-efficacy leading to an improved ability to advocate for good software practices.

The participants’ self-efficacy was strengthened through the knowledge and skills shared within the fellows’ community. For example, Cara commented: “what the SSI gave me was a bit of an exposure that, actually, I could be better, I could learn new skills, and I could share those skills, again, more broadly using the tried and testing routes I already had.” Confidence for advocacy was supported through experiencing a sense of belonging in the SSI with Theresa saying that “within the SSI, I think, yes, actually, here, I fit, and I have that confidence to go out and spread that message and get more people involved.” The status of being an SSI fellow also benefited some participants’ confidence, as well as broadening their networks. For example, Sophie commented that “saying I’ve got an SSI fellowship has given me a lot of confidence and capital… to approach deputy directors, team leads, directors, director generals who I would never have spoken to” and Cara describes how “people would ask me my opinion as my contribution to things. So, I would say my standing’s increased and the diversity of projects I’m involved with has increased.” The Fellowship increased some participants’ capabilities to work independently with Cara saying that “it [the Fellowship] was … the first time I did something outside of my research group that was me, and clearly me and me alone, it wasn’t a by-product of someone else I worked with.” Similarly, some participants gained the opportunity to develop leadership qualities. Vicky commented that “So, I’m now more involved with the reproducibility agenda at the university, more widely, which I would attribute, at least partly to my SSI fellowship.”

Interview finding 2: The Fellowship generated positive outcome expectations of engaging with the SSI community and alignment of to the fellows’ professional goals with their advocacy work.

The Fellowship strengthened the participants outcome expectations from their fellowship activities through building a community with shared goals and promoting collaborative activity. For example, Frank saw his SSI work as a community project, and it was the collaborative aspect of which he was most proud. Additionally, being awarded a Fellowship promoted a commitment to engage with the SSI which then led to positive outcomes. For example, Vicky described attending an SSI Collaborations Workshop as part of her fellowship which she would not have done otherwise, and this seeded a successful joint funding bid with others at this event.

Allowing the participants to adapt the plans they submitted as part of their application allowed them to align their wider professional goals with their advocacy work. For example, Theresa had difficulties delivering her plans due to changing role, but she was able to take advantage of opportunities for advocacy that came through the SSI. The mutuality between supporting career progression and advocacy is also seen with Cara stating that the SSI “want their fellows to be successful because then they advocate in more varied and higher places. And I definitely feel that I have succeeded in that element.” Additionally, some participants spoke about their continued engagement with the SSI after their inaugural period, demonstrating their long-term alignment with the SSI’s goals. For example, Adrian noted that “yes, absolutely, I do still feel like an SSI fellow… whenever I talk about research data as an output, I always include code and workflows and protocols and all the sort of stuff that is the SSI’s, kind of, bread and butter.”

Case studies

Additionally, we developed a case study for each of the six participants, titled as follows: 1) Vicky: Supporting emergent leadership in research reproducibility; 2) Theresa: Enabling the transition into research policy; 3) Sophie: Developing domain-specific community leaders; 4) Cara: Developing university-wide experts in software sustainability 5) Frank: Developing expertise in improving software good practices; and 6) Adrian: Advocating for software in research policy roles. These have been presented in the detailed report [4].

General finding: areas for improvement

Three key areas for improvement within the fellowship programme emerged from the research. The first is participants’ lack of confidence in the impact of their advocacy activities within their institutes and/or domains. This uncertainty may reduce their ability to leverage the fellowship for promotions and job applications. For those fellows with non-standard academic career paths, where the traditional markers of research success are not always relevant, this is of particular importance. Additionally, it could limit the SSI’s ability to demonstrate the efficacy of the Fellowship to funders and to strengthen its reputation within the academic community.

The second area for improvement is the mentorship component. While some participants reported beneficial mentor relationships, others had little to no contact. Several also noted insufficient guidance for both mentors and mentees. The final area concerns time buy-out. Some respondents experienced tension between their substantive roles at their home institute and the fellowship with unclear expectations around the time commitment. Notably, this tension sometimes stemmed from participants’ enthusiasm towards the fellowship. In other cases, participants relied on supportive managers to allow time to pursue work related, but distinct from, their primary role.

Recommendations

To support others to develop similar programmes, we propose the following recommendations along with a model of the Fellowship (Figure 1) based on the Theory of Change [6]. The recommendations are evidence-based, as they were derived from the interview data, and are theoretically informed by applying Social Cognitive Career Theory [8] during the analysis. We have also included recommendations based on the participant-reported areas for improvement.

First, to foster growth in self-efficacy, which in turn supports increased impact, we recommend the following: provide an online forum (for example, Slack) for fellows to share their knowledge, skills and test project ideas; invite applicants from multiple career stages, domains and career paths; highlight the professional status of programme alumni to enhance the programme’s reputation; publicise new Fellows to enhance their visibility and open opportunities; and highlight the diversity of career routes and roles amongst fellows and staff to cultivate a strong sense of belonging particularly for fellows who have non-standard paths. Second, to strengthen the fellows’ outcome expectations from their advocacy activities we recommend the following: offer a wider community forum for initiating collaborative projects, (for example, the SSI Collaborations Workshop); provide a stipend to remove reliance on institutional approval for workshops and training; and establish lifelong fellowships to enable continued engagement as fellows’ roles evolve. Finally, to allow the fellows to align their professional goals with advocacy work we recommend the following: adopt a flexible approach to fellows implementing their plans and invite them to participate in institutional activities (for example, campaigns and recruitment of future fellows).

Further research is needed to develop evidence-based solutions for the areas requiring improvement. However, based on our understanding of the Fellowship, we propose the following enhancements: implement an impact evaluation framework with a standardised reporting mechanism to provide clearer insights into fellows’ contributions to their institutes and domains; formalise the mentorship component by developing guidelines on purpose and structure and providing training for mentors (this has now been implemented for the 2025 cohort); and enable fellows to employ graduates and assume supervisory roles to reduce the tension caused by the absence of time buy-out. A final recommendation comes from the positive reports that the fellowship supported some participants to overcome barriers related to underrepresentation in academia, such as gender, disability, career stage, or being a first-generation graduate. We suggest broadening this support for other underrepresented characteristics, such as race and sexual orientation.

Conclusions

In this article, we analysed whether the SSI fellowship supports its fellows to become advocates for good software practice. Survey responses demonstrated that, for most participants, the Fellowship strengthened their knowledge and use of software practices and supported their professional development. However, they were less confident in stating whether their fellowship activities had achieved their intended aims within their institutes and/or domains. We suggest this could be resolved with the use of an impact evaluation framework that would support fellows to understand and record the impact of their work. It would also provide data through which the SSI could evidence the programme’s benefits.

The interview data, analysed using Social Cognitive Career Theory [8], demonstrates that increased self-efficacy enhanced the participants’ ability to advocate for good software practices. For example, some participants noted a growth in professional status from having ‘the badge’ of SSI Fellow, as well as through the independence and leadership qualities they built during their Fellowship projects. Some participants also described an increase in confidence through finding a sense of belonging at the SSI. Additionally, the data shows that participants developed positive expectations of collaborating within the SSI’s networks on advocacy related work, and there was an alignment of the participants’ professional goals and activities promoting good software practices. This indicates that the programme fosters a multidisciplinary network of research software related professionals who have the knowledge and confidence to advocate for good software practices and the potential to improve research quality within their institutes and domains.

Using the interview data we also identified the key mechanisms through which the participants gained these positive outcomes. These mechanisms include: building a community of fellows around a common goal by providing a platform for peer-level knowledge sharing; encouraging collaborations among fellows, institute staff and external partners; supporting diversity in career stage, career pathway and domain; providing a flexible stipend that avoids the need for institutional approval; offering personalised support through institute staff and mentorship while encouraging autonomy and leadership; and granting lifelong fellowships. By identifying these mechanisms and addressing areas for improvement, as well as sharing a model of the Fellowship (Figure 1), we provide practical recommendations for those designing similar advocacy or career development programmes.

Competing Interests

In this article we evaluate the impact of the Software Sustainability Institute’s Fellowship programme. However, three of the four authors (AB, CJ, SS) were employed by the SSI during this research.

Author Information

Anita is a Research Associate based in the Department of Computer Science. Her current research focus is using social science methods to understand and improve equity in computational research careers. She also has a PhD in medical image analysis and has worked as a research software engineer.

Aishah is a Student Success Coordinator in the Directorate for the Student Experience. She has a social science research background in inclusive research careers.

Shoaib is a Project Portfolio Manager in the Department of Computer Science. He is the Fellowship Lead at the Software Sustainability Institute, and a researcher with the European Life Science Infrastructure (ELIXIR) and the European Open Science Cloud (EOSC) on research software quality practices.

Caroline is a Professor in the Department of Computer Science. She is also Director of Research at the Software Sustainability Institute and Head of Research for the Department of Engineering at the University of Manchester.