Introduction
Arguments for and against the use of Artificial Intelligence (AI) in all fields of life are rife with polarised views, ranging from the belief it could pose an extinction level threat to arguments that it could be a remarkable force for good (Tomasev et al., 2020; Tredinnick and Laybats, 2023; Stahl and Eke, 2024). Within higher education (HE) these debates are mirrored – on the one hand AI could be used unethically, to gain unfair advantage, thus undermining the integrity of university qualifications and research (Sharples, 2023; Yusuf et al., 2024). However, there are many who believe it offers opportunities to enhance student learning and support, create more authentic and inclusive assessment and support new forms of research (Michel-Villareal et al., 2023).
The emergence of these polarised positions coincided to some extent with concerns about the future of traditional examinations in HE, particularly for distance learning institutions, such as the University of London and the Open University UK. These and other open and distance providers were forced to move the classic model of face-to-face examinations in examination centres, to online equivalents during the pandemic. Subsequently, internal concerns, including increased assessment irregularities and potential grade inflation, coupled with external pressures, for example, from Professional Statutory and Regulatory Bodies (PSRBs), have resulted in institutions moving some assessments back to traditional formats utilising examination centres and invigilation.
In this paper, which draws on a workshop held at the EDEN conference in Dublin, in summer 2023 and subsequent workshops at the University of London in the autumn of 2023, we seek to share an account of early attitudes of practitioners and students to the affordances and disrupters presented by AI and online assessment.
Between 2020 and 2023, the University of London (UoL) conducted a longitudinal evaluation of online assessment in its online and distance programmes, focusing on a range of stakeholders and factors, including student experience and student outcomes, feedback from the examiners and programme directors, and from operational teams. Student sentiment survey results were broadly positive with support for a future move to online assessment with very little concern about student performance or reputation of qualifications being negatively affected. Interviews with academic programme directors between 2020 and 2023 have revealed a varied picture of reform of assessment design, to include more continuous assessment and review and redesign of assessment modes and content. At the same time, although pass rates have increased there is evidence that so, too, have academic offences, leading to concerns at institutional level about academic integrity (Hatzipanagos et al., 2020; Amrane-Cooper et al., 2023a).
Over a similar period, the Open University, UK (OUUK), which has been conducting a comprehensive university-wide review of academic conduct since 2021, initiated a (unpublished) literature review on academic integrity in 2022 and explored the future of assessment in international workshops (Rossade et al., 2022). An institutional survey of students from over fifty modules explored the distance learners’ experience of, and opinions about, online remote exams and exam preparation and compared this to data collected about conventional exam-taking before the COVID pandemic (Cross et al., 2023). The data report similar findings and outcomes to those of the University of London, also raising an additional question about potential variation of experience by student age.
The issues identified by both institutions have been further impacted by the arrival of AI tools, and particularly Generative AI. Concerns about how assessment can and should be redesigned to mitigate against the potential challenges presented by AI generated text, and to incorporate its affordances have emerged (Bower et al., 2024; Jisc, 2024). There is, thus, a set of questions emerging from the possibilities of employing online assessment and AI in the assessment practices of the universities and their partner institutions for which there are no easy answers, and which deserve wide and urgent discussion in the HE sector.
In this research we set out to explore the perceptions of practitioners and students. We focussed on the challenges and opportunities presented by AI and online assessment when creating authentic, meaningful and inclusive assessment. Although our primary focus is online and distance education, the spread of online and hybrid education_highlights the importance of a shared understanding of the challenges that new AI tools bring to the realm of assessment. . To explore this, we held three workshops within a five-month period. We approached this investigation by asking participants to consider two future scenarios and, in this paper, we present the outcomes from the workshops and consider the efficacy of our approach.
Background
Theory and practice of online assessment in recent literature has been reconfigured under the two disruptors we discussed earlier – the pandemic and AI. Some of the recent optimism comes from the development of assessment platforms that deal with assessment design and proctoring, offering a supposedly robust solution for the future. However, assertions such as Adzima’s (2020) claim that online assessment provides the same opportunities to commit an academic offence as in face-to-face learning environment has played out. Furthermore, online assessment appears to have become problematic in the post generative AI tools era.
The literature on the relationship between assessment and AI in HE reveals a rapidly evolving field, characterised by both enthusiasm and caution, as highlighted in the introduction. There are promising advancements combined with growing concerns (Cardona et al., 2023) Research emphasises the potential of AI technologies that attempt to streamline assessment processes, improve personalised feedback, and enhance scalability and efficiency. However, criticisms and concerns about the future also highlight important issues of failings in equity, privacy, and the need for human oversight (Bozkurt et al., 2023), e.g. in maintaining the integrity and validity of assessments.
While AI offers innovative solutions to longstanding challenges in HE assessment, careful consideration of its implementation and ethical implications remains imperative for fostering inclusive and effective learning environments. Van Wyk et al. (2023) explored the views of 13 academics on the use of ChatGPT as an AI based learning strategy, in an open /distance learning context. Their findings reflected the polarisation cited in our introduction and highlighted the concerns about academic integrity which the use of such AI tools appears to present. Students’ attitudes toward online assessment and AI are diverse. While some embrace online assessment for reasons of convenience and flexibility, others express concerns about its reliability, privacy, concerns for technical issues and the potential for biases and impact on reputation of degrees (Amrane-Cooper et al., 2023a). Similarly, attitudes toward AI in assessment vary, with some students appreciating its potential to support student learning, while others fear a loss of human touch and raise concerns about data privacy and ethical implications (Jisc, 2024). Overall, students’ perceptions are influenced by factors such as digital literacies, digital access, educational background, behavioural intention, trust in the assessment system, and personal beliefs about the role of technology in education (Amrane-Cooper et al., 2023a; Strzelecki, 2023).
Methods
As already acknowledged, the future of AI and its use in HE in rapidly evolving. Much of the speculation, referred to in our introduction, identifies the potential risks and benefits to universities and their communities, including students and future employers. The problem that we face in trying to manage those risks and benefits lies in lack of certainty over what the future will be like, thus creating challenges in approaching research. Both Van Wyk et al. (2023) and Yusuf et al. (2024)elicited views from participants through requests via a Microsoft Teams site and discussion board respectively. Van Wyk et al. note that previous reports in this field have used quantitative methods and advocate the use of more qualitative work or mixed methods. The developments we consider here, however, are taking place in a context characterised by complexity and ambiguity as institutions struggle with policy and funding challenges. Workloads are heavy and finding time to learn about and trial new tools is not easy. For these reasons we adopted what Ross (2018) refers to as speculative methods and defines as “research approaches that explore and create possible futures under conditions of complexity and uncertainty”. In later work, referring specifically to the field of digital education and learning, she talks about working in a critical or questioning way requiring “methods that can bring particular ideas or issues into focus by envisioning or crafting conditions which may not yet currently exist, working to trouble established imaginaries’”(Ross, 2023, p13).
Speculative methods can take the form of scenario building or working back from a future imagined state. Bozkurt et al. (2023) for example, used fictional story telling as an approach to elicit views on what living with AI in the future might be like. These resulted in both dystopian and utopian visions. In our research participants were asked to imagine both negative (‘hell’) and positive (‘heaven’) futures in which generative AI tools become ubiquitous in university settings, specifically in the context of online assessment. In our speculative approach we invited participants to respond to two scenarios, using the following prompts:
Are concerns about academic integrity and the advent of AI tools deterring innovation in online assessment and driving the sector back to inauthentic assessment approaches such as classic invigilated exam centre modes?
What opportunities do online assessment and AI tools provide to enable more authentic, meaningful and inclusive modes of assessment for learning? What might these models look like?
Three separate workshops were held, firstly face-to-face with delegates at the 2023 European Distance and E--learning network (EDEN) conference (Amrane-Cooper et al., 2023b), followed by hybrid workshop at the UoL Centre for Online and Distance Education and, finally, an online student workshop at the University of London. The workshop design aimed to provide space for debate and discussion on the specific intractable or difficult problem of AI and online assessment and sat well in relation to speculative approaches.
At the EDEN workshop there were 16 participants initially working in 4 groups but reduced to 3 for the final future state ‘imagining’. The participants were all delegates at the conference and held academic and professional service roles at a range of international universities. They were asked to consider what a future hell state could be like if our worst fears about generative AI and online assessment were realised. They considered what a future heaven state could be, if there were no constraints, for example of regulation or resource. Their ideas and views were collected on sticky notes and transcribed.
One limitation of this approach (particularly for the EDEN workshop) lay in drawing from a single pool of conference attendees where it was not possible to collect any personal data (particularly job role) which might influence the way they thought about the subject. They were self-selected so clearly had an interest and appeared excited about the opportunities and concerned about the risks.
The second workshop followed a similar pattern, but with a larger number of participants, 18 in person and 13 online, and more information about them was available to us. In the room, participants were drawn from academic roles in education, business, technology, and law. There were also colleagues who have roles in learning technology within their contexts and those whose focus is student support. Online, the participants were more evenly drawn from learning design/technology roles and academics with disciplines such as politics, diplomacy, biology, and veterinary science represented. Across both groups, colleagues from a range of positions, including executives, senior managers, independent consultants and emeritus Professors were present. All but one were UK based and they represented 9 UK HE institutions. Again, participants were asked to envisage ‘heaven and hell’ scenarios, using sticky notes in the room, and an online collaborative tool for the online participants.
The thematic analysis (Braun and Clarke, 2022) was carried out, by the four authors, working in pairs, in online workshops. Firstly, each pair considered the commonalities and differences in the ‘heaven’ scenarios and recorded these in an online collaborative tool, identifying key words in the process. The exercise was repeated for the ‘hell’ scenarios and both groups then came together to discuss the emergent sub-themes. Two tables were then constructed to amalgamate the ‘heaven’ and ‘hell’ outputs from both groups which resulted in the emergence of key themes, as discussed in the next section.
However, it was clear from the data that an important voice, that of students, was missing. A third workshop was thus held on-line with volunteers from UoL and OUUK. Nine University of London students, a mixture of Bachelor and Master levels from both within and external to the UK participated and one OUUK student was present. The structure followed the same pattern as the staff workshops, as did the manual thematic analysis with the additional data being added to the existing online collaborative tool and tables.
Although the facilitators observed rich discussions in all three groups, time was a major limitation. Ideally the workshops would be extended over at least a day, allowing more depth and better ways of capturing the outputs. Using sticky notes and online collaborative tools to capture thoughts and ideas was somewhat limiting in that any nuances of discussion were sometimes lost in the summarising. Furthermore, responses could not be attributed to specific roles (other than staff or student) and interpretations of the outputs was constrained due to brevity in some cases.
Findings and discussion
The workshops revealed a balance of views across staff groups about both challenges and opportunities for managing the future of assessment post-pandemic and in the light of rapidly developing AI. Despite the limitations of the approach, discussed above, the authors were able to identify clear themes, broadly replicated in all three workshops. However, participants appeared to find it challenging to separate out ‘heaven’ and ‘hell’, resulting in negative responses mixed into the positive views and vice versa.
It was also notable that the same broad themes were reflected in all workshops with the student outputs generally being more specific and concrete, reflecting a potential concern about the impact on them as individuals. For example, where staff talked about a lack of integrity undermining the qualification, students expressed concern about the advantage that other students may gain through dishonest means.
We identified four key themes: ethics and assessment integrity, the necessity for the redesign of assessment, diversity and inclusion, and dependencies. We explore these themes below, drawing on our data and present some characteristic viewpoints.
Ethics and assessment integrity
In both discussions with staff there was a focus on the issue of academic integrity and the threat AI poses to the institutional reputation. For example, staff expressed concerns that Professional Statutory Regulatory Body (PSRB) recognition, which adds value to the student qualifications in the employment market, may be removed unless the regulators and awarding bodies are confident that online assessment is secure. Comments included ‘loss of credibility’ and suggestions that “universities would lose accreditations with professional bodies”. A question was raised about control and whether this should lie with the professional bodies or the HE sector, but the general tenor was that professional recognition was important.
The broader concern, around loss of credibility, linked to the potential award of credentials without clear demonstration of competence was discussed in the staff and student workshops and included discussion of both online assessment and AI. An allied concern stems from the idea that generative AI tools can potentially allow unethical behaviour to be rewarded. This was particularly evident in the student workshop which included discussion of the idea that others may gain an unfair advantage: “AI can support students who haven’t done any work”. Students also expressed a concern about the implications of undetectable use of AI tools: “AI outperforms and no matter how hard you work, the cheating student, using AI, gets the better mark”. Furthermore, concern was raised that a student might be incorrectly accused of misconduct due to the lack of effective AI detection tools. Conversely, a student commented that AI itself could be an excellent tool for detecting misuse of AI in assessment because: “It can detect to high precision whether a student is cheating – AI is able to become the perfect guardian of integrity (like Turnitin)”.
Staff groups also believed that developing effective integrity monitoring could counteract the existential threat AI might present for universities. Nonetheless, a more fundamental risk was articulated as the potential breakdown of values and integrity which devalues HE. As one participant commented: “If students use AI without limits or checks, it could devalue the degree. Is it [the degree] worth anything?” It was also argued that “time spent focusing on possible cheating and time spent policing” moves attention away from learning and fears were expressed that “students using generative AI tools to produce assignments would not appreciate that they were no longer learning”.
Redesigning assessment
The pressure to implement changes to assessment was identified by some participants as a reaction to recent developments, such as the experience of online assessment during the pandemic and the emergence of widely available AI tools. However, it was noted that a positive aspect of these developments was that they added an “incentive to redesign”, with potential for “improvement in authenticity” and student experience. Student perspectives and attitudes were considered as valuable, and staff participants acknowledged that it would be beneficial to “involve students in assessment design”.
According to staff participants, some of the positive developments presented by AI tools could include the automation of repetitive tasks: e.g. “supporting automatic marking and feedback”. The opportunity for implementation of “personalised learning at scale” was also noted as an enabler for the development of scalable authentic assessment. However, the difficulties of “implementing authentic assessment at scale” and failing to design assessments that relate to future real world and work-related experiences were also raised.
However, an increase in proctored online exams was seen as a possible, short-term compromise to some of the challenges. There were concerns expressed that a return to in-person invigilated exams and traditional assessments would undermine gains and initiatives achieved through redesigning assessment. This could lead to “loss of innovation” and of much that had been achieved in taking forward assessment for learning. Other concerns included lack of flexibility and sticking with “online assessment without improvement”.
The reasons for such potential negative developments were discussed in all three workshops, and some example sentiments included concerns that the security of online assessment would lead to a return to invigilated in person exams with pen and paper and the push for undergraduate exams in exam centres following professional body requirements. It was argued that moving towards more continuous assessment was put at risk by returning to face-to-face examinations, i.e. shifting the balance between continuous assessment and final proctored exams.
Diversity and inclusion
Discussion in the staff and student workshops included a significant focus on issues of equality, diversity, and inclusion in both the heaven and hell scenarios. Looking first at the positive outcomes within a heaven scenario, the potential of AI to support more equitable outcomes for students was noted by staff and students. For example, staff noted “translation AI tools support assessment tasks – [making assessment] more inclusive”, whilst the student group discussed the potential for AI to support accessibility for students who have difficulty with typing, noting that “improvements to text to speech are possible”. Staff considered the link between learning and assessment in considering diversity, as illustrated by the viewpoint that the “affordances of AI such as translation, summarizing, can assist more diverse students to focus on higher order assessment outcomes”.
The opportunity for more equitable assessment was discussed by all groups. Staff noted the potential for AI and online assessment to support “fair/objective assessment”, and this view was supported by the student participants who suggested “AI could objectively grade closer than human markers” and who noted the opportunities for AI involvement to remove marker bias. However, student workshop participants also raised concerns about AI being biased in grading and in online proctoring scenarios – “bias to groups of students, for example their names” and “bias of video recording to skin pigmentation”. Students were further concerned over the potential inequities arising from AI being used in the creation of assessment submissions. Discussion focussed on the potential for a student using AI to get the better mark, no matter how hard the system tries to block this: “Is this fair to those who have worked really hard?”.
The discussions in each workshop also focussed on digital inequalities with students and staff noting that online assessment and access to AI require finances, stable electricity, Wi-Fi, and therefore present inequity. However, the student groups noted the opportunity for improved outcomes for all through access to fast, effective, responsive tutoring facilitated by AI.
Dependencies
In light of the possibilities offered by fast developing future of assessment, a range of dependencies were identified for the successful engagement with both AI and the post-pandemic adoption of online assessment practices. In this section we discuss those entities whose success is dictated by another entity or resource.
Firstly, there was a clear view that students and staff will need support and development if assessment is not to move backwards in terms of learning, teaching and pedagogy, to return to unseen exam papers in exam halls. One viewpoint was that there were “powerful pressures reinforcing summative assessment only”. In particular, it was argued that assessment design will need to be revised to mitigate legitimate concerns about risks to academic integrity. Staff would need professional development in the design of authentic assessment, and students would need to be prepared also for change in their understanding of assessment methodology. Our data indicate that some of this development might be valuably undertaken on a collaborative basis, with students and staff working together. Innovation in assessment design will need to be promoted, embedded and supported both within disciplines and at an institutional level. It was suggested that there was an “opportunity to assess students using open-ended creative questions that require research and synthesis”. Of particular concern are the resources of time and energy available to staff after the demands of the online pivot during the pandemic emergency, as well as other intense pressures. Additionally, it was recommended that institutional assessment policies needed to be urgently reviewed and further developed to reinforce positive outcomes for students.
Conclusion
In our workshops, we used the initial binary scheme of ‘heaven’ and ‘hell’ to highlight the potential for radical improvements through online assessment and generative AI in learning and teaching—represented by the metaphor of ‘heaven’—as well as the possibility of significantly poorer outcomes, symbolised by the metaphor of ‘hell’.
This methodological approach effectively served as a heuristic tool to encourage reflection on the significant moment that generative AI was in the process of creating. Our focus was less on theory building than on developing understanding to support the necessary development of practice. Some participants in our study argued that the potential for change embodied in AI surpassed that of the digital revolution that began over 30 years ago. However, as we evaluated the evidence collected at the workshops, we realised that the ‘heaven-hell’ dichotomy we had set up was not accepted by a number of workshop participants, who saw a future of ambivalences, dependencies, contingencies and ambiguities as generative AI and online assessment were further promoted and adopted. The complexity of the future with threat and opportunity closely interconnected has all the characteristics of a ‘wicked problem’, where solutions are hard to imagine as well as to deliver.
Nevertheless, HE, and particularly the online and distance sector, have become adept at building on and benefiting from technical and pedagogical disruption. Our findings indicate that there were similar concerns arising for staff and students, concerns about ethics and assessment integrity, about the pulls and pushes on assessment design, about equality and inclusion, and about the complex dependencies required to move forward positively. These findings validate the caution found in existing literature, pointing to a need for oversight to mitigate academic misconduct in AI enhanced assessments.
The concerns raised by students and staff echo literature on digital literacies and access disparities in HE (Amrane-Cooper et al., 2023a; Strzelecki, 2023). In the light of the complex set of findings from the three workshops, therefore, we propose establishing some priorities for inclusion in both policy and practice as online learning and AI continue to transform the landscapes of learning, teaching and assessment:
Issues of digital inclusion will continue to demand close attention, as will the needs of students, teaching and professional staff for professional development and training. This was evident across all the four themes identified. New tools have the potential to enhance teaching, learning and assessment but can further embed existing disadvantage or even create new inequalities. Finding the time and resource for staff to undertake professional development and ensuring this is available for students, as well as taking into consideration their views, will be vital.
HE providers should embed and deepen their focus on the purpose, context and values of assessment rather than being technology-led. A deeper and shared understanding of what academic integrity means, especially for students, may mitigate some of the potential pitfalls and contribute to a more authentic experience of assessment as (rather than ‘for’ or ‘of’) learning. This call for value-driven assessment, prioritising outcomes that reflect meaningful student growth and development, echoes ongoing debates about academic integrity and the necessity for authentic learning experiences (Cardona et al., 2023; Bozkurt et al., 2023).
We should recognise that AI can present us with the opportunity to reduce administrative labour, in order to focus on creativity and originality but at the same time we should recognise that concerns remain over bias. Furthermore, if AI continues to produce variations that align with prevailing opinions or trends, it might create a narrow view of what is possible, reducing the likelihood of considering alternative or innovative options. AI as an efficiency enhancing tool can reduce anxiety and support diversity but our findings indicate that it must be used in the knowledge that limitations exist, aligning with Van Wyk et al.’s (2023) findings.
These three points in summary make clear that the HE sector, including its online and distance institutions, faces a steep climb over the foreseeable future in integrating AI into learning, teaching and assessment in terms of understanding, equity and workload. This climb cannot and should not be avoided, and we hope that our work, by situating our findings within the broader theoretical conversation, will serve to underpin its most effective and equitable development, and to support change in practice for the sector.
Data accessibility statement
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Ethics and consent
Ethical approval was not applicable for this study, as the data were collected from publicly accessible events, such as a conference and academic fellows’ meetings, without any collection of identifiable personal information.
Competing interests
The authors have no competing interests to declare.
Author contributions (CRediT)
Amrane-Cooper: Conceptualization, methodology, formal analysis, investigation, data curation, visualization, writing—original draft preparation, Writing – original draft, Writing – review & editing. Stylianos, Hatzipanagos; Conceptualization, methodology, formal analysis, investigation, data curation, visualization, writing—original draft preparation, Writing – original draft, Writing – review & editing. Liz, Marr: Conceptualization, methodology, formal analysis, investigation, data curation, visualization, writing—original draft preparation, Writing – original draft, Writing – review & editing. Alan, Tait: Conceptualization, methodology, formal analysis, investigation, data curation, visualization, writing—original draft preparation, Writing – original draft, Writing – review & editing. All authors have read and agreed to the published version of the manuscript.