A Systematic-Narrative Review of Online Proctoring Systems and a Case for Open Standards

Online Proctoring System Features Mapped Against Examination Steps

In the traditional approach to taking exams in higher education, students use pen and paper as tools and sit in large examination halls, supervised by human proctors. The proctors check student identities and monitor students to prevent access to unauthorized material and interactions with others. Computer-based proctoring systems were introduced in the early 2000s (Foster & Layman, 2013) with the aim of allowing students to take exams in their own environments using their own computers while still ensuring assessment integrity. Today’s systems use a variety of technologies to address the tasks of identity checking and monitoring and combine automated steps with human intervention.

Before exam start, the identity of the student needs to be verified against pre-recorded or live evidence. This can be as simple as asking the student to present an identity card, such as a student id, to a camera, followed by live verification through the remote human proctor. Alternatively, pre-exam authentication can be based on sophisticated biometric models created before an exam session and used for automated authentication (Labayen et al., 2021; Okada et al., 2019). The physical environment of the student is inspected via a video room sweep to make certain that only the student is present. The computer system of the student is secured, e.g., with a lockdown browser, to ensure the student only has access to approved resources.

During the exam, proctoring monitors if the student complies with exam rules and does not receive external help. In a more traditional approach to online proctoring, a human proctor monitors the student via video feeds and decides if an intervention is required. In newer systems the monitoring task shifts to the computer system. Using technologies such as face or voice detection, the system checks if an additional person has entered the student’s physical space or if the student engages in a conversation. Some automated systems intervene directly in the test process, alerting the student, e.g., to adjust their camera. Exam sessions are recorded, capturing data streams from video, audio, screen, keyboard and pointing devices for post-exam analysis. These different approaches are described as live, automated, and recording (e.g., see Arnò et al., 2021; Hussein et al., 2020; Labayen et al., 2021; Nigam et al., 2021) and work in combination. The proctoring data are analysed after the exam to determine if rule violations have taken place.

To strengthen authentication and monitoring further, suggestions have been made to combine data from multiple exam sessions. For example, Fidas et al. (2023) propose to store biometric data or handwriting samples from multiple exam sessions to be used for comparisons in subsequent exams. In addition, data derived from non-exam related student interaction with the learning management system can be used to strengthen model development (Fidas et al., 2023; Whitelock et al., 2020). This results in registration and profiling phases which relate to contexts and time periods outside specific exam sessions. Table 1 provides an overview and adds information on the involvement of human proctors and computer systems.

The Nature of Proctoring Data

Compared to traditional assessment systems, proctoring changes the nature of data collected. While personal identifiers (name, date of birth, student id, …) are required for any online system, proctoring with new technologies adds physiological (face and voice characteristics, fingerprints, …) and behavioural (typing patterns, mouse movement patterns, eye movements, …) biometrics. While student images are already commonly used, e.g., as profile pictures in learning management systems, registration and authentication require higher resolution images. During the monitoring and review phases, online proctoring creates data on potential rule violations and on the decisions made in response. While there are some parallels to plagiarism detection, online proctoring data do not focus on the assessment material but on physical and behavioural characteristics of the student. As discussed by Whitelock et al. (2020) and Laamanen et al. (2021), students are concerned about the use of such personal data. Compared to manual proctoring, using automated and AI processing creates the capacity for near endless data collection and analysis, and large datasets are required for the training of AI systems (Aurelia et al., 2023; Fidas et al., 2023). While AI-enabled or AI-based proctoring is now common, many authors (e.g., see Khanna et al., 2021; Motwani et al., 2021; Nigam et al., 2021; Slusky, 2020; Tweissi et al., 2022) do not describe how AI is used. Table 2 compares the data used by advanced proctoring technologies to those traditionally used with assessment technologies.

Data Privacy and Security in the Online Proctoring Literature

While several reviews of online proctoring systems have been published over the last years, the coverage of data privacy and security is limited (see Table 3). Arnò et al. (2021) and Labayen et al. (2021) comment on the GDPR (General Data Protection Regulation) compliance of the systems investigated yet provide little detail on how such compliance is addressed or what it means in the context on online proctoring. While Nigam et al. (2021) write about the importance of data privacy they do not address privacy concerns in their descriptions of proctoring systems. Aurelia et al. (2023) take a cheating prevention perspective and do not address potential student concerns. While some authors (Hussein et al., 2020; Whitelock et al., 2020) write about trust, they focus on establishing trust into the integrity of exams conducted online and not on the trust a student can have into the system chosen by their education provider.

Open-Source Proctoring Systems

As stated by Foster and Layman (2013) and more recently Arnò et al. (2021), it is challenging to uncover details of online proctoring systems. Most systems are commercial products with proprietary licencing, preventing access to details on system architecture and code. The two system reviews that addressed licencing found one open-source system (Hussein et al., 2020) which is no longer available and an open-source lockdown browser (Arnò et al., 2021) which is still available but does not constitute a full proctoring system.

The TeSLA system discussed by Okada et al. (2019) and Whitelock et al. (2020) promises valuable features, including e-authentication instruments based on biometric and textual analysis tools. After initial research funding from the European Commission, the development into a commercially available system was envisaged. Yet, current details could not be found at the time of writing. Labayan et al.’s (2021) online proctoring systems review serves as justification for development of the SMOWL system. The authors refer to SMOWL’s data privacy policy that separates biometric data from identity data when creating the biometric model for a student by indirection via a separate identification code. SMOWL has since been developed into a commercial system. Fidas et al.’s (2023) TRUSTID system is described as ‘online student identity management system’ (p16), available open source in the TRUSTID code repository as proof-of-concept version. The system uses a client server architecture with a server-side backend and two client applications. A smartphone app is used to capture face and voice data for the creation of biometric user profiles. A desktop application is used during exams to verify the student’s identity and monitor for exam rule violations. The TRUSTID architecture puts strong emphasis on the protection of student data across storage, retrieval and processing and the authors discuss mechanisms such as encryption, distributed ledger technologies (often referred to as blockchain technologies) and digital wallets. TRUSTID aims at providing students with control over their data following principles of self-sovereignty. For example, students control access to the biometric models stored in the smartphone app to which they grant their university access on request. The last activity on the TRUSTID code repository occurred in July 2023 and a video system demonstration created around the same time is available.

Research Objectives and Questions

The review presented in this introduction shows that online proctoring draws on physiological and behavioural student characteristics that are compiled into sophisticated data models, created and accessed by AI tools. Information on the treatment of data privacy and security is limited as is the insight into the AI algorithms employed. This situation is concerning for many reasons. Swauger (2020) points to the potential for discrimination against non-traditional students and a pedagogy of punishment based on allegations of cheating, propagated by technology companies who are in line for big profits. Coghlan et al. (2021) outline challenges regarding transparency, the reliance on companies meeting legal and ethical requirements, and the lack of real choice for students. Students have no alternative to consenting to proctoring (McKenna, 2022) regardless of concerns about the invasive nature of online proctoring. Institutional policies have not kept pace with new forms of technologies and data collections (Brown & Klein, 2023).

The years of the COVID-19 pandemic have seen a rushed adoption of online proctoring technologies (Selwyn et al., 2023). Having experienced many challenges (e.g., concerns about privacy and equity, tests marked as zero) universities are switching to a more thoughtful adoption of online proctoring and system selection (e.g., my university has suspended the use of online proctoring for a year to carefully evaluate). While this includes deliberation on the need for exams, prominent researchers continue to see exams and online proctoring as important components of university education (e.g., see Coghlan et al., 2021; Dawson, 2020; Najar & Ahmad, 2024; Nicola-Richmond et al., 2024; Selwyn et al., 2023). To contribute to a responsible adoption of online proctoring and system selection, I am undertaking a new review of online proctoring systems focused on data privacy and security and AI. My research questions are:

1. Who are the providers of online proctoring systems, are open-source systems available?

2. How do online proctoring system providers address data privacy and security?

3. How do online proctoring system providers address AI use?

4. How might higher education institutions select online proctoring systems responsibly?

Online Proctoring Systems Providers

As noted in previous reviews, online proctoring is dominated by commercial providers. All 33 systems reviewed are commercial services (the open-source version of MOODLE PROCTORING is an incomplete solution). As explained in the introduction, I have not been able to locate current versions of the TeSLA and TRUSTID systems. Despite substantial research funding and cooperation across universities, it seems challenging to develop full systems. The systems with a recent higher education research context (DUGGA, SMOWL, SYNAP) are all commercial services. With company locations in Asia, Australia, Europe and North America, providers are spread across many geographical areas. This aligns with the challenges in securing assessment integrity felt globally.

While several systems primarily focus on the provision of proctoring services, others foreground testing and certification services (e.g., SURPASS and YOUTESTME). The focus of some system websites is less on explaining the online proctoring options than promoting wider assessment services (e.g., INSPERA PROCTORING), switching to online versus on-site exams (e.g., DIGIEXAM), or learning management system adoption (e.g., UXEPERTISE XP). The varied company foci add to the challenges of determining the level of uptake of a proctoring system in the higher education context, something I attempted based on the clients named on the websites.

The answer to my first research question is therefore that online proctoring is firmly in the hands of commercial providers from education/assessment and testing/test centre backgrounds. I could not locate a comprehensive open source online proctoring solution.

Data Privacy and Security

The system websites vary widely in how they address data privacy and security. While DUGGA and MOODLE PROCTORING/PRO do not address the topics at all, some proctoring companies build on a pedagogy of punishment and allegations of cheating as described by Swauger (2020). For example, DIGIPROCTOR foregrounds students’ sophistication in cheating, and TALVIEW writes about cheating and impersonation to emphasize the need for proctoring services. Others (e.g., CONSTRUCTOR PROCTOR, PROCTORU, SURPASS) provide access to generic policies that do not mention proctoring specific data, are written in legal language, and are not displayed prominently. On the other hand, several providers follow the values plauded by Coghlan et al. (2021) by addressing students, explaining data privacy and security in accessible language, making it easy to find this information, and showing concern for the needs of students. For example, EXAMITY, HONORLOCK, INTEGRITY ADVOCATE, PROCTORIO and ROSALYN write in a style that addresses students directly. ROSALYN and TALVIEW foreground the information in easily accessible trust sections. PROCTORTRACK provides access to a dashboard showing what data have been collected and when those are scheduled for deletion and facilitate data deletion requests.

Twelve systems address GDPR compliance at various levels. This reaches from brief references and generic information (e.g., DIGIEXAM, PROCTOREDU) to meaningful explanations of the concepts of data controller and data processor in the context of online proctoring (KRYTERION). Some systems provide information on the location of the data centres used and the implications regarding country specific regulations (e.g., HIREPRO, HONORLOCK), the length of data storage (e.g., IRIS INVIGILATION), or the range of individuals who can access student data (e.g., PROCTORIO). Most systems address data security to some degree in the information provided via their websites. Topics include encryption (e.g., EXAMONLINE), hardware and operational security (e.g., TALVIEW), backup and disaster recovery (e.g., INSPERA), auditing (e.g., SYNAP), ethical hacking (e.g., PROCOTORIO) and security features integrated into cloud service provisions (e.g., TESTWE).

These observations lead to a two-level answer for my second research question. First, some system providers demonstrate little care for data privacy and security. Education institutions should carefully evaluate if choosing these providers aligns with their values. Second, despite the care for data privacy and security demonstrated by several providers, many questions remain unanswered. One area of concern relates to the types of data related to online proctoring. Systems construct student profiles to strengthen pre-exam authentication and confirm student identity throughout the exam session. This raises questions about consent and ownership. Students should be active participants in contributing to the development of their proctoring profile, for example in choosing when they permit the extraction of biometrics for their identity profile. Students should have access to this profile, be able to facilitate additional data collection, and control its use. We should draw on the concept of data sovereignty, described as ‘meaningful control, ownership, and other claims to data or data infrastructures’ (Hummel et al., 2021, p12) and discussed by Grech et al. (2021) as self-sovereign student identities in the context of academic records.

The meaningful application of GDPR is another area of concern. GDPR contains the concepts of data controller and data processor. The data controller decides the purposes, the ‘why’, and the means, the ‘how’, of the collection and processing of personal data; the data processor carries out the instructions of the data controller (European Commission, n.d.). In proctoring, the educational institution is the data controller and the proctoring company the data processor. Yet, as Giannopoulou et al. (2023) conclude in their review of legal decisions regarding online proctoring and GDPR, those lines are blurred, e.g., regarding the processing the proctoring companies perform on the data collected. I question if educational institutions in their role as data controller have enough insight into the processing steps to control the ‘how’ aspects.

AI Use

Following the earlier discussion on automated processing and decision making (automated flags, alerts, decisions), my focus now shifts to the use of AI. Of the 24 systems with descriptions comprehensive enough for closer inspection, two thirds refer to the use of AI in supporting data processing and decision-making tasks (some, e.g., INTEGRITY ADVOCATE and IRIS INVIGILATION, instead talk about detecting student use of generative AI to compose answers). There are references to the use of AI in analysing data to assist with authentication and alerting to potential rule violations, yet little detail on specifics is provided. Some statements are difficult to assess without further explanation (e.g., CONSTRUCTOR PROCTOR’s ‘1,000,000+ Video & audio in AI-trained datasets’). On some websites it appears that references to AI are included to create the impression of advanced system capabilities. Others refer to the increasing importance of AI technologies and their ‘sharp awareness of its evolution’ (KRYTERION).

Two systems stand out from the others. TALVIEW offer a new AI proctoring mode based on Alvy, a ‘dedicated AI Proctoring Agent, revolutionizing how exams are monitored’. TALVIEW claim that ‘Alvy can identify nearly 90% of the events recognized by both human proctors and the traditional AI model.’ The difference between Alvy and TALVIEW’s automated proctoring mode seems to lie in Alvy’s direct interaction with students. Differences in terms of data analysis are not outlined. ROSALYN stands out by the level of detail provided in language accessible to the end user, e.g., on candidate profiling, gaze and speech analysis (see this blog entry as example). ROSALYN also emphasizes the need for the continuous development of AI models and explicitly references the use of proctoring data for the training of those models.

The answer to my third research question is that we are lacking meaningful insights into the use of AI in online proctoring systems. As Selwyn et al. (2023) state, education institutions have little insight into the functioning of proctoring systems. It is questionable if institutions can look ‘under the hood of the software’ and undertake ‘due diligence’ as necessary according to New Zealand’s past Privacy Commissioner (Walters, 2021, no page number). A recent systematic review on the use of AI in education by Forero-Corba and Negre Bennasar (2023) lauds the potential of AI techniques to improve education. The challenge is to find a way forward that fulfils Giannopoulou et al.’s (2023) demand for educational technologies that reflect the ethos of education institutions.

One question to be addressed is how much technical detail should be available to give adequate insights. Somavarapu et al. (2024) provide a review of AI technologies used in research projects focussed on online proctoring solutions. The authors name the attributes (e.g., face detection, face alignment, head pose estimation) and the specific AI technologies used (e.g., SURF, Viola-Jones and SIFT algorithms for face detection). Details on attributes are valuable to a more general audience such as educators and students. Among the systems I have reviewed only ROSALYN provides comparable details. While restricted of interest to a technical audience, details on algorithms and machine learning approaches should be available and could build on ‘model cards’, introduced to increase transparency on machine learning and AI technologies (Mitchell et al., 2019).

Beyond the level of technical detail, I also seek more clarity on the level of autonomy AI has in proctoring. While AI is already widely used for analysing data, building models and making predictions, it appears that most systems do not yet allow for autonomous decisions without human supervision. This needs to be clarified, especially considering the rapid technological advances in AI technologies and their increasing levels of adoption. This also applies to generative AI. For now, TALVIEW’s AI proctoring agent Alvy appears to be the only generative AI component. Undoubtedly other systems will follow.

Towards the Responsible Selection of Online Proctoring Systems

My fourth research question asks how education institutions might select online proctoring systems responsibly. As demonstrated by the information provided in this article, there are big differences in how system providers see issues around data privacy and security and attitudes towards assessment integrity. After pre-selecting a system provider who values the protection of student data and foregrounds student interests, institutions can use the overviews on the proctoring steps, data and systems (Tables 1 and 2, Appendix A) to seek further clarifications. Somavarapu et al. (2024) review advanced technologies (e.g., face alignment, face spoofing detection, eye gaze estimation, open mouth detection) and the underlying algorithms. While I agree with the authors that online proctoring systems should draw on technological advances, we need to focus on how those technologies are used in responsible ways. Models supporting the selection of educational technologies and their implementation in complex higher education environments, as discussed by Chugh (2023), provide valuable guidance that can be adopted for the proctoring system context. AI policy frameworks highlight the pedagogical, operational, and governance dimensions to be considered (Chan, 2023). Selecting the right system will go a long way towards responsible use of online proctoring in the immediate future.

Considering the dominance of proprietary proctoring systems and the challenges observed in maintaining and further developing open-source approaches, a potential path forward lies in the creation of open standards for data collection, management and exchange.

Limitations of this Research

The review of the proctoring systems presented in this research is based on the descriptions provided by the companies offering those systems. This was appropriate as the research design included looking at the perspectives on data privacy and security as portrayed by the companies. Yet, it is a limitation as an independent test of the systems could have provided better insights into their functionalities. Exploring all systems hands-on would have been challenging as independent access is not available and their proprietary nature prevents studying the algorithms and technologies used. Staff at the educational institutions conversing with system providers might gain additional information. Yet, given the proprietary settings, the depth of the new insights is likely to be limited. Students are unlikely to gain access to the more detailed information.

Future Work

In their explorations of a right to digital education Celeste and Gregorio (2023, p250) conclude that ‘digital education has been left in the hands of private actors whose main objective is the pursuit of economic gains’. This sentiment rings true for the online proctoring space where system provision is dominated by commercial providers. I call on educational institutions to regain control and to work with students on finding just solutions. While the ultimate goal would be to create leading edge open-source proctoring systems, the history of the systems such as TeSLA and TRUSTID (no longer available despite substantial research efforts) as well as MOODLE PROCTORING and SMOWL (transition from research-based to commercial systems) demonstrate the challenges in achieving this. Having commercial systems adhere to open standards would go a long way in building solutions that facilitate transparency and interactivity.

The importance of open standards in ensuring quality and interoperability of learning technology products and services has been acknowledged (Koper, 2008). Yet, there are many challenges in definition and pathway to adoption (Duval & Verbert, 2008). While online proctoring has its specific challenges, there is common ground with other learning technologies in areas such as continuous authentication, which is tasked with providing ‘unobtrusive and continuous user authentication, behind the scene, and without inconveniencing the users’ (Soltani et al., 2021, p6), or the work on decentralized self-sovereign digital identities that provide verifiable credentials (e.g., see Sedlmeir et al., 2021). The work towards open standards in the online proctoring space needs to link to such initiatives.