Figure 1
Diagram defining real world learning in the context of assessment. Framework promoted by Archer, Morley & Souppez (2021) Real World Learning and Authentic Assessment https://link.springer.com/chapter/10.1007/978-3-030-46951-1_14 (CCBY).
Figure 2
Series of flags describing different roles in terms of AI and assessment design. Own work (CCBY).
Table 1
Examples of approaches, related to specific discipline, to advance data and AI literacy. Own work (CCBY).
| DISCIPLINE | ASSESSMENT MODEL | FIRST YEAR UNDERGRAD LEVEL 4 | SECOND YEAR UNDERGRAD LEVEL 5 | THIRD YEAR UNDERGRAD AND POSTGRAD LEVEL 6 AND 7 |
|---|---|---|---|---|
| Computer Science | Project-Based Learning | Apply basic programming skills to simple datasets. | Develop more complex algorithms for data analysis. | Innovate and optimise algorithms, demonstrating advanced programming skills. |
| Business | Case Study Analysis | Identify basic business data trends. | Analyse and interpret complex business datasets. | Critically evaluate business data, considering ethical and strategic implications. |
| Social Sciences | Research Paper | Discuss social implications of basic AI usage. | Explore the societal impact of AI in specific contexts. | Investigate and analyse complex social issues related to AI, applying critical perspectives. |
| Health Sciences | Group Presentation | Present basic health data insights. | Communicate findings on health-related data, using relevant tools. | Articulate advanced health data insights, considering ethical implications. |
| Engineering | Problem-Solving Exercise | Apply simple AI solutions to engineering problems. | Use intermediate AI techniques to address engineering challenges. | Employ advanced AI methodologies for innovative engineering solutions, evaluating their impact. |
| Environmental Studies | Real world Data Project | Collect and analyse basic environmental data. | Use advanced data analysis for complex environmental projects. | Engage in comprehensive environmental data projects, addressing critical ecological issues. |
| Humanities | Data Ethics Reflection | Reflect on basic ethical considerations in humanities research. | Analyse and reflect on ethical implications of data use in humanities contexts. | Demonstrate a nuanced understanding of data ethics in humanities, proposing ethical frameworks. |
Table 2
Prompts for assessing proficiency in data and AI literacy for learners. Own work (CCBY).
| CRITERIA | NOVICE | INTERMEDIATE | PROFICIENT |
|---|---|---|---|
| Understanding Data Concepts | Limited understanding of basic data concepts. | Solid understanding of fundamental data concepts. | Advanced understanding of data concepts, including data types, sources, and formats. |
| Data Analysis Skills | Struggles to analyse and interpret basic data sets. | Capable of conducting basic data analysis and drawing simple conclusions. | Excels in advanced data analysis, employs statistical methods, and derives meaningful insights. |
| AI Awareness and Understanding | Limited awareness of AI concepts and applications. | Clear understanding of AI principles and basic applications. | In-depth understanding of advanced AI concepts, including machine learning algorithms and their practical use. |
| Critical Thinking in AI Context | Struggles to critically evaluate AI outputs and implications. | Demonstrates the ability to critically assess AI models and outputs. | Excels in evaluating AI models, considering ethical implications, and proposing improvements. |
| Communication of Data and AI Insights | Struggles to communicate insights coherently. | Communicates data and AI insights effectively. | Presents complex data and AI findings clearly, using appropriate visualisations and language. |
| Data Ethics and Justice Considerations | Limited awareness of ethical considerations in data and AI. | Recognises basic ethical considerations and their importance. | Demonstrates a deep understanding of data ethics and justice, actively considers and addresses ethical concerns in data and AI practices. |
Figure 3
Series of rhombi describing the levels of power and arrows showing how it manifests. Source – Understanding the levels at which power operates in the collection and use of data, and how it manifests – Data Justice in Practice: A Guide for Impacted Communities (CCBY).
Figure 4
Pillars of data justice in relation with authentic and real world assessment design. Adapted from Taylor (2017) – Own work (CCBY).
Table 3
Examples of authentic assessment design and implementation through the use and with the support of specific AI systems. Own work (CCBY).
| TYPE | DATA JUSTICE PILLARS | EXAMPLE | CRITICAL DATA AND AI LITERACY |
|---|---|---|---|
| Scenario Based | Engagement with technology | Academics could investigate the notion of introducing (or having their students enter) the parameters of their assessments into technologies like ChatGPT and asking it to generate a real world brief by acting as a ‘client’ in the context of their discipline. ChatGPT, or another GenerativeAI system, was requested to play as a client for a marketing firm and develop a brief for a marketing expert to construct a campaign for an undisclosed product, including a budget, timeframe, and market reach. When students enter their evaluation settings, GenAI system develops a personalised task for them and generates a unique situation each time. It also helps increase the legitimacy and applicability of the evaluation (Nerantzi et al., 2023). | Competencies related to the selection of specific input data to introduce in the GenAI system but also the competence of critiquing and interpreting the data produced by the GenAI itself (Ng et al., 2021). |
| Students were introduced to a research proposal assessment that required them to propose a scenario-based research challenge and create a research study. Students were given an AI-generated study proposal, which they then reviewed in groups before sharing their findings with the rest of the group. A further benefit is the ease with which instructors may employ AI to create instructional tools. The goal was not just to deploy AI technologies to help students consolidate discipline-specific abilities, but also to illustrate the advantages and disadvantages of specific GenAI systems (Nerantzi et al., 2023). | Competences related to “work with, analyse, and argue with data as part of a broader process of inquiry into the world” (D’Ignazio, 2017) and “communicate and collaborate effectively with AI” (Long & Magerko, 2020, p. 2). | ||
| Case Studies | Engagement with technology | Using AI as a thought partner in the development of branching scenarios to provide learners with realistic critical thinking evaluations. Creating case studies can be time-consuming, and existing ones may lack essential components for a successful learning experience. Using AI, we can create not just a complete case study, but also pertinent discussion, feedback, and branching that lead the learner on a content-rich journey (Nerantzi et al., 2023). | Development of critical thinking assessment skills, promoted through the use of AI as a tool to create real world authentic scenarios, in a safe formative online environment (Cui et al., 2023). |
| Students choose a real world example of how AI has changed some element of practice, such as voting in a political election, financial decision making, parole judgements in law, or medical diagnosis. They then examine the ramifications and repercussions of the case, assessing the role it plays in occupations that may be related to their discipline but require specialised application. They can also identify some essential talents or attributes that they may need to enhance or develop in their current work or potential future career paths (JISC, 2023). | This specific activity could support the development of competences such as critical evaluation, AI literacy (e.g.ethics and data protection) and metacognition (Ashford-Rowe, Herrington, and Brown 2014; Mohamed & Lebar, 2017; Ng et al., 2021). | ||
| Human VS AI written assessment | Engagement with technology Visibility | Encourage students to analyse an article written by a person vs one created by a GenAI system. Ask students to review the GenAI output and provide instances of statements that appear plausible but are incorrect or do not make sense. Stimuli students to verify the sources and references. Ask them to identify any gaps in GenAI’s coverage and to share their views regarding the terminology introduced. The primary goal is to assist learners to strengthen their critical analytical abilities by making judgements (Nerantzi et al., 2023). | Development of data and AI literacy competencies related to the possibility of accessing, critically evaluating and using data sources (Prado & Marzal, 2013; Shields, 2005). |
| The goal is for students to question the authenticity, correctness, or applicability of ChatGPT (or other GenAI system) replies to their assessment titles, and then use Google to determine where the information came from. This then leads to a fact-checking exercise in which they verify or expand on what ChatGPT proposes. As a result, they just consider ChatGPT as a beginning point which leads to serious academic study. This might be done immediately, asynchronously, or as a preliminary stage before beginning, such as an annotated bibliography or text analysis exercise (Nerantzi et al., 2023). | “Encourage learners to investigate who created the dataset, how the data was collected, and what the limitations of the dataset are. This may involve choosing datasets that are relevant to learners’ lives” (Long & Magerko, 2020, p.6). | ||
| Project based | Non discrimination | Students develop a product that answers a real-life problem, e.g. practical solution for engineering or computers, a professional development template for a business student. Having completed a draft, they submit it to peers and/or stakeholders for comment. Students may use AI to create ideas, improve presentations, seek guidance on component selection, and more. They create a reflective narrative (or exhibition) to support their product, which details the design process, decisions taken, teamwork, and stakeholder participation. Realistic management of the scope of work and available resources is required (JISC, 2023). | This activity design could support the development of students’ competencies related to the sphere of metacognition, research, practical competence, assessment literacy, collaboration and they can become able to collect and critically analyse the data produced from AI (Sadler, 1989; Long & Magerko, 2020). |
| Students received an assignment, such as writing a policy evaluation of food security reform in a global context or curating an exhibition on a topic covered in their course. They pick their topic by the third week of the module. Early in the module, participants submit/present brief drafts (with instructions on what these should include) for comments from staff and students. AI may be used to produce ideas, which can then be questioned, changed, and merged as needed. They receive three pieces of input on the draft and they submit and present a final draft but do not receive comments on areas that they had the opportunity to submit and gain input on earlier in the semester (JISC, 2023). | This formative design can sustain the development of students’ competencies connected to research, planning, general/key professional and assessment skills, assessment skills, problem solving and AI literacy (Sadler, 1989; Ng et al., 2021). |