Figure 1
Summary of challenges in teaching and learning spoken English (Burns 2019; Ferlazzo 2023, Leong & Ahmadi 2017; Nartha 2024; Rivera & Villanueva 2023).
Figure 2
Partnership agents for co-creating AIED.
Figure 3
The DBR Process Model as practiced in the current pilot.
Table 1
Phases of the DBR model in practice – Iterations and design decisions.
| ITERATION NO. | FINDINGS AND CHALLENGES | DESIGN DECISIONS FOR THE NEXT ITERATION |
|---|---|---|
| 0 Stage | (Preparatory Phase) Researchers compiled a list of specialized AI-powered platforms currently available. Researchers finalized study structure, workshop schedule, and design pattern elements. Pilot participants were selected and contacted, and their participation confirmed. | Three specialized platforms were chosen to test during the pilot. Option was left open to test general-purpose GenAI. |
| Iteration 1 | (Meeting 1) Teachers tested one of the preselected, specialized platforms (Platform X) and filled out a questionnaire noting their experiences, feedback, and questions. The research team shared these findings with the platform developer. (Meeting 2) Teachers tested a general-purpose GenAI platform using a pre-written prompt and shared their experiences in group discussion. Teachers learned the design pattern (DP) framework and created their first design pattern (DP1, Part A) using an AI platform of their choice (specialized or general purpose). | (Following Meeting 1) Research team reviewed and analyzed teachers’ questionnaire responses and analyzed suitability of the specialized platforms. One of the specialized platforms was determined unsuitable for matriculation preparation. (Following Meeting 2) Teachers refined DP1 (Part A) individually or in pairs, ran it in class, and wrote their feedback (Part B). Research team reviewed and analyzed data from DP1 and identified important criteria, including ease of use, onboarding techniques, platform feedback accuracy, student engagement, level, and rubric alignment. The general-purpose GenAI platform was determined unsuitable for matriculation preparation. |
| Iteration 2 | (Meeting 3) Teachers viewed a live demo and then tested Platform Y and provided feedback to the platform developers. Teachers began a second design pattern (DP2) in groups. | (Following Meeting 3) Teachers refined DP2 (Part A) individually or in pairs, ran it in class, and wrote their feedback (Part B). Research team reviewed and analyzed data from DP2. Most criteria identified during DP1 were reconfirmed. Technical problems with the platforms were identified. A second specialized platform was determined unsuitable for matriculation preparation. Developers of the remaining specialized platform refined the platform according to teachers’ feedback and requests for rubric alignment. |
| Iteration 3 | (Meeting 4) Teachers watched a demo of Platform X (revised) and then retested and provided feedback to the platform developers. (Meeting 5) Teachers responded via group discussion to their trials with Platform X (revised). Teachers retested the platform in pairs, assessed each other’s performances as if it was the real exam, compared their own assessments with that of the platform, and recorded their comparisons in writing. Teachers were asked to suggest changes to the platform assessment rubric. (Meeting 6) Teachers were asked to rank the design principles extracted from the previous iterations and to justify their selection from their own professional point of view and the inferred perspective of their students. | (Following Meeting 4) Teachers ran a sample oral matriculation exam using Platform X. Teachers first graded their students’ results as they would have the actual exam and then compared these results with the AI-driven platform feedback. (Following Meeting 5) Research team reviewed and analyzed data collected from the teacher trials during Meeting 5. (Following Meeting 6) Research team reviewed and analyzed data collected from the ranking during Meeting 6. |
Table 2
Platforms tested during the pilot.
| PLATFORM CHARACTERISTICS | GENAI-POWERED PLATFORM |
|---|---|
| General-use GenAI application allowing voice recognition with an option to request feedback on the processed text (speech-to-text). | Bard |
| General-use GenAI application with voice recognition plugin. | ChatGPT (v.3.5) |
| A customizable avatar with a human-like appearance, capable of conducting a dialogue based on generative AI technologies and a customizable system response setting. The platform includes separate dashboards for teachers and students. | BV (Specialized platform for spoken English) |
| Simulations of scripted conversations presented by human actors in short video clips. Learners first view a brief interaction between two characters and are then asked to play one of them and speak directly to the other, using a written script for guidance. A simple point system rewards accurate speech, and teachers have access to student data. | SP (Specialized platform for spoken English) |
| Scenarios of English-language conversations with an avatar, including simulations and activities. The system is adaptive, identifying the learners’ level and adjusting itself accordingly while providing a detailed analysis of various assessment metrics, including a recording of the learner’s speech, with separate dashboards for teachers and students. | SN (Specialized platform for spoken English) |
Table 3
Design principles for GenAI-based platforms for preparing for the matriculation exam in spoken English.
| PRIMARY CATEGORY | SECONDARY CATEGORY | DESIGN PRINCIPLE | AVG | ST | EXP. CITATIONS |
|---|---|---|---|---|---|
| Pedagogic Alignment | Content level | Vocabulary practice/enrichment | 4.6 | 0.91 | Vocabulary is the foundation of language. Having prompt words on the screen in ‘describe a picture’ activities is great! |
| Real-life or meaningful topics | 4.3 | 1.12 | Yes! Crucial so they stay motivated, curious and enjoy it. This is super important for students to be engaged in the speaking activities. | ||
| Personalization (in real time/responsive) | 4.2 | 0.72 | Personalization was important to my students. If this means that the AI avatar says the student’s name, YES! People love that! | ||
| Teacher-chosen content | 4.2 | 0.75 | Teachers should be able to choose the right content for their classes. | ||
| Attributes | Provides a broad choice of tasks | 4.8 | 0.69 | Students feel empowered and motivated when they have a choice. A variety of choices gives the students freedom to choose what meets their needs and interests. Too much choice can be overwhelming. | |
| Encourages repetition/asking questions | 4.2 | 1.12 | Asking questions is important – repetition less so – as the students can do the activity again | ||
| Allows open-ended/free conversation | 4.1 | 1.06 | Necessary to give our students as close to a ‘real life’ simulator as we can. This scares me as a teacher … I can imagine students trying to draw the AI into inappropriate topics. | ||
| Encourages taking initiative in conversing | 3.65 | 1.27 | You can easily figure that it is a robot [at] work and, in most cases, responses are automatic… but this characteristic in particular [makes] my students give up in some cases and feel frustrated. This is good as long as it is clear to the students what they need to do. I think it’s nice when it reflects human conversation. | ||
| Scaffolding | Makes use of prompts (in English or L1) | 4.4 | 0.45 | Great for weaker/intermediate students, but also for stronger students who just need the prompt to re-focus their attention. | |
| Allows teachers to create or alter prompts | 3.7 | 1.06 | Many teachers want the system ready to use. We [teachers] like it when things are ready. | ||
| UX/UI | Navigation | Activity length indicator | 4.22 | 0.83 | Important! Time management for both the students and teachers to either plan the class time accordingly, or for the students to know their pace (especially for COBE time limit, practice managing their speaking time). It is very important for the student to see his conversation length. |
| Option to sort or choose activities by Bands (Note: Lexical Bands are CEFR-based but specific to Israel) | 4.1 | 0.75 | Students don’t really care, and we should not care that much as long as they are speaking. Excellent idea for specific vocabulary practice or to suit the class’s level. CEFR levels will make this useful [also] for people outside of Israel. | ||
| Onboarding materials/ideas for teachers | 3.88 | 0.94 | Get[ing] the teachers on board, comfortable with using it first, then use it in class- from experience [this] is key for it to work [in the] long term. If it isn’t simple, they won’t use it. | ||
| Lexical Band indicator on each activity | 3.78 | 0.83 | I would like a toggle option from the teacher’s side. | ||
| Onboarding materials/ideas for students | 3.75 | 0.94 | Yes – I spent too much time explaining to the students what to do. I found students often skipped it. | ||
| Accuracy | Accuracy of speech interpretation | 4.75 | 0.47 | [The ability to recognize] different accents (or just bad pronunciation). Very important for fluency, understanding. | |
| Performs even in noisy environment | 4.33 | 0.37 | Using the AI tools in a large class can be really problematic. …otherwise, I will not be able to use it in class. | ||
| Accessibility | Avatar relatability/likeability | 4.5 | 0.5 | Especially [important] the likeability. Most avatars did not match the age group of my students. Important – maybe use avatars of different identities and allow students to choose/change from time to time. Needs to seem human. | |
| Platform access from home | 4.44 | 0 | Although many pupils only work at school, those who want to continue should be able to. Especially important on phones. Good for homework. Some schools lack a computer lab. | ||
| Assess- ment | Criteria | Rubric criteria completely parallel COBE | 4.22 | 0.47 | Would be nice, though the most important thing for students is to communicate, but there could be more COBE focused activities/feedback. There should also be easier leveled rubrics that the teacher can choose from. |
| Practice features completely parallel COBE | 3.89 | 0.94 | Not all speaking is COBE preparation. It’s nice for students to have different options, including fun activities, not just COBE questions. | ||
| Confidence assessment | 3.33 | 1.25 | It would be a great indicator Fluency is more important. Important, though when we want to encourage weak learners, it is not the most important thing. | ||
| Only positive feedback/reinforcement | 2.3 | 1.04 | If it is only positive, it won’t really help. On the other hand, if opportunities for learning are phrased in a positive way, that’s the best. Depends on the level of the students…Ensure that struggling students receive as much positive feedback as possible | ||
| Format | Teacher access to student activity | 4.6 | 0 | This will enable the teachers to observe their students’ performance and check their progress. Great for teachers to get a full understanding. Yes! I know my students the best… (sorry…) | |
| Instant/real-time feedback for students | 4.5 | 0.69 | Important for them and encourages them to try again. | ||
| Instant/real-time feedback for teachers | 4.4 | 0.34 | Important to know who is working, how to help, mediate etc… So we can give our own feedback as well. Reports are more important than instant feedback for teachers. | ||
| Variety of assessment feedback formats (not just visual) | 4 | 1.21 | Visual, written, audio, translated into Hebrew/Arabic… Definitely – I highly recommend…an oral feedback format. Yes, end of the term reports on vocab learned, for example, or a ‘certificate of excellence’ auto generated can go a long way! |