Lessons Learned: Creating Tutorials to Teach Agent-Based Modelling to Archaeologists

2.1 Open educational resources

The ABMA project sought to create Open (Open Access and Open Source) Educational Resources (OERs) for training in Agent-Based Modelling for Archaeologists. These OERs were to be HTML- and JavaScript-based so they could be used on any device with a browser, even if it is not connected to the internet. The full programme consists of five hands-on practical tutorials, taking learners through the process of creating digital models. The modular architecture ensures that they can be independent or incorporated into other teaching tools like learning management systems. They are also deliberately ‘method agnostic’ so they can be incorporated into any tools from self-teaching to MOOCs, and are supported by a set of materials to encourage use of the OERs.

The activities that the project implemented included:

• development of a hands-on vocational training programme, aligned with the recently published textbook Agent-Based Modelling for Archaeology: Simulating the Complexity of Societies (Romanowska, Wren and Crabtree 2021);

• conversion of the training programme into modular learning resources that are interactive, based on HTML and JavaScript standards;

• creation of support materials to encourage use of the OERs;

• creation of how-to guides to help trainers incorporate OERs into their teaching;

• creation of a code and learning materials repository that will allow the project to ensure sustainability after the European Union support has ended and to facilitate the creation of an open-source community to manage and improve the OERs in the future;

• development of a framework for skills acquisition aligned with the Digital Skills Passport;

• promotion through a series of multiplier events.

2.2 Objectives and target audience

The expected results and outcomes, for the target audience of professional archaeologists, were:

1. a surge in professionals who can build, apply, and critique archaeological simulation, thus opening new research avenues for the discipline as a whole;

2. increased level of digital competence, specifically in advanced computer skills that are highly transferable to other industries. This includes technical skills such as programming, but also “soft” skills related to computational thinking;

3. improved opportunities for archaeologists to work in other European countries (transnational mobility) through matching the skills being delivered with the European Commission’s Digital Competence Framework for Citizens (2.2) and the sectoral Skills Passport for Archaeology, which means that it will be easier for individual workers to demonstrate their skills to employers across Europe;

4. increased opportunities for professional development by having access to professional training that can be incorporated into formal or non-formal/informal learning.

For the secondary target audience of vocational training providers in archaeology, e.g. employers, colleges and not-for-profit organisations, the expected results and outcomes were:

1. increased quality of training in Agent-Based Modelling that is free and open;

2. access to teaching materials for skills that are in high demand and aligned with employers’ needs.

We were able to draw on previous experiences with a small private online course (SPOC) on ‘Modelling and Simulation in Archaeology’ that some of us designed for graduate teaching at the Faculty of Archaeology, Leiden University, where we taught this course in 2016/17 and again in 2018/19 (Scherjon, Romanowska and Lambers 2019). For this course we developed new online teaching materials comprising pre-recorded lectures, practical exercises, reading assignments and exams. While we greatly benefited from that experience, the target group of the SPOC was graduate students with prior experience in digital archaeology, whereas the ERASMUS+ project was aimed at a much broader target group of archaeology students and professionals whose computational literacy could not be assumed.

2.3 Online tutorials

The project team developed five online tutorials. The tutorials were first designed using storyboards and later converted into interactive guided walk-throughs using NetLogo Web (https://www.netlogoweb.org/launch) and JavaScript and hosted on GitHub (https://github.com/ABMArchaeologists/ABMA_tutorials). During the project, initial testing was done by all authors and the issues functionality in GitHub was used to tackle bugs.

The ABMA learning materials assume that participants have learning skills at least at a secondary education level. Also, they are suitable for participants with at least some background in archaeology, such as those typical among students after the first year of a Bachelor study in Archaeology, i.e., above EQF levels 4 or 5. In addition, the tutorials are aimed at professionals working in archaeology who have some experience with computer applications. We also assume that learners are proficient in English and have at least the Reading B1 level, but Reading B2 is recommended (Council of Europe 2020). The tutorials were also developed with a secondary aim to develop learners’ general skills in the following competence areas: 1. Information and data literacy, 2. Communication and collaboration, 3. Digital content creation and, 5. Problem solving (European Commission, Joint Research Centre et al. 2022).

After initial debugging by the project team, the tutorials were tested on a wider sample of participants during several online and in-person events, all of which were run by members of the project team (Table 1). Recognising the difficulty of delivering practical training in an online environment (Microsoft Teams and Zoom) during the online events, several members of the team were present which enabled participants to go to break-out rooms with an expert if they needed additional help or wanted to discuss things. Throughout the project we sought feedback from the participants and used it to improve the tutorials prior to the following events. The feedback was obtained through two surveys using Qualtrics (https://www.qualtrics.com/). The participants of the events were asked to respond to one survey before the start of the event and another one after the end of the event. The first survey focused on the background of the participants and their original level of knowledge in relation to ABM (see appendix 1 for the questions). At the end of the event the participants were asked to answer a second survey. Questions of this survey were aimed at measuring the effectiveness of the tutorials and getting feedback on the workshop and tutorials (see appendix 2 for the questions). For some events participants had to register beforehand and we were able to send the pre-workshop survey by email. At other events no registration was possible or necessary and the pre-workshop survey was shared at the start of the event. The post-workshop survey was distributed using QR codes or links at the end of the event.

The surveys were analysed using R (R Core Team 2023). The following packages were used for analyses: ggplot2 (Wickham 2016), dplyr (Wickham et al. 2023), tidyr (Wickham, Vaughan and Girlich 2023), forcats (Wickham 2023a), lubridate (Grolemund and Wickham 2011) and stringr (Wickham 2023b). For the sake of reproducibility (Marwick 2017) this paper was written in R Markdown (https://rmarkdown.rstudio.com/), with data and code available at https://github.com/ABMArchaeologists/ABMA_paper (Visser et al. 2024b).

Over the course of the project three groups of students worked on the project during the Smart Solutions Semester at Saxion University of Applied Sciences. This is an interdisciplinary semester in which students of at least three different study-programmes or disciplines work together on a complex problem/project (https://www.saxion.edu/business-and-research/collaborate-with-saxion/smart-solutions). Within the semester students have to come up with new ideas or solutions to help a client with a wicked, or at least challenging, problem. The backgrounds of the various students we worked with were diverse: Applied Computer Science, Archaeology, Business Management Studies, Creative Business, Creative Media & Game Technologies, and ICT. The aim of involving interdisciplinary groups in the project was to involve students in current research projects, to enable them to learn new things and share their innovative ideas and solutions with us. The outcomes of the student groups were supplementary to the deliverables of the project, but their new ideas were beneficial to the project. Their help during workshops was useful, but they also were engaged in developing educational material (Aalpoel et al. 2024), testing the tutorials, and developing a style for the website (Jutte et al. 2024) and other materials.

2.4 Dissemination of tutorials

The ABMA tutorials are publicly hosted on GitHub (https://github.com/ABMArchaeologists/ABMA_tutorials/) and are citeable (Rocks-Macqueen et al. 2024). Anyone interested can download the tutorials and their source code. To increase the accessibility and visibility of the tutorials, they are also hosted on a static website (https://abmarchaeologists.github.io/ABMA_website) and can be followed without the need to install any software. The source code of the website is also openly available (Jutte et al. 2024).

Besides the tutorial, the website provides a link that can be used for feedback, basic explanations of agent-based modelling, an introductory presentation from one of the workshops, videos from an online course at Leiden University (Scherjon, Romanowska and Lambers 2019) and external resources that people can use to learn agent-based modelling. The website was mainly developed by a student group from Saxion University of Applied Sciences (Jutte et al. 2024).

3.1 Tutorials

The final set of ABMA teaching materials comprises, next to accompanying materials (see above), the following tutorials (Rocks-Macqueen et al. 2024):

• Tutorial 1: Introduction to ABM

• Tutorial 2: Beginning with NetLogo

• Tutorial 3: Expanded ABM skills

• Tutorial 4: Intermediate ABM

• Tutorial 5: How to Model

Each tutorial consists of a number of individual lessons, which guide the learner in a self-paced manner through an increasingly challenging set of exercises. Collectively, they build the skills and knowledge necessary to become a confident agent-based modeller.

In the first tutorial the users learn what simulations and agent-based models are and how they can help in archaeological research (Figure 1). This tutorial consists of four lessons and makes use of the Artificial Anasazi model (Axtell et al. 2002). The first two lessons introduce the learner to simulation in general and Agent-Based Modelling in specific. Various concepts related to ABM, such as the definitions of model and simulation, are introduced and the participants take their first steps in NetLogo syntax. The third lesson explains how ABMs are used in archaeological research. The final lesson introduces the learner to the NetLogo Interface and the difference between the Interactive and Authoring mode. This tutorial builds a foundation for working in NetLogo and with ABM. All activity stays within the entry level of difficulty, although some higher-level concepts are lightly touched upon.

Figure 1

In tutorial 2 the user learns the basics of NetLogo by building their first own simulation modelled on the Out of Africa dispersal of Homo sapiens (Young and Bettinger 1995). The learner works through the basics of NetLogo syntax and learns how to set up a simulation and visualize its outcomes. This tutorial consists of nine lessons that guide the learner from the introduction to the beginner level of proficiency. The learning curve is relatively gentle. This is achieved by working with the NetLogo web interface that enables highlighting specific words and buttons and giving visual cues as to how to proceed. The tutorial introduces several commonly used built-in functions in Netlogo (called primitives) and introduces the general structure of all models consisting of the initialization phase and the main simulation loop. It walks the participants through the most important features of the NetLogo world, such as dimensions, coordinates and the origin point of the grid on which the simulation runs. The learners gain their first stripes in programming by using simple loops and functions as well as conditional statements. The next level is achieved by introducing the topic of function definition and different types of variables. Exporting data produced by the model through plots is also explained.

In the third tutorial, the learner builds a simple trade model, again based on a published simulation (Romanowska 2018). Here learners expand their NetLogo programming skills with more complex syntactic structures such as loops, lists and reporters. They are introduced to some key computational techniques like modular code development and debugging, which become important alongside an increased coding complexity. This tutorial consists of seven lessons. It intends to guide the learner from the beginner to the intermediate level of proficiency. This is achieved by introducing standards of code development such as modular code, pseudocode, debugging and annotating the code. In addition, the tutorial introduces custom agent breeds, visualization with labels and reporters, plots and monitors. This advances the proficiency level of the learner, since it develops knowledge of the programming language combined with diagnosing unexpected behaviour and problem solving.

In tutorial 4 the learner works with the seminal Sugarscape model (Epstein and Axtell 1996) to further expand their agent-based modelling and NetLogo skills. The learner explores how to set up more complex interactions between agents and the environment. Furthermore, the principles of setting up good experiments and validating models are explained. This tutorial consists of eight lessons. It intends to solidify the learner’s understanding at the intermediate level of proficiency by introducing yet more complex agent-environment interaction, alternative ways to visualize the environment and modelling spatial dynamics. Creating toy landscapes is central to lesson 5. The learner dives into the topic of experiment design at a practical level but also with regards to the scientific principles, such as validation. Thus, they are guided through the process of setting up experiments in NetLogo and collecting results using monitors and plots. The topic of validation of agent-based models is critical for most archaeological applications and thus several ways to compare simulation results with the archaeological record are explored.

In the fifth and final tutorial, more is learned about how to incorporate agent-based modelling in archaeological research. This tutorial focuses less on programming in NetLogo and more on the principles and standards of computer-based modelling. The tutorial consists of five lessons. It summarizes the material delivered in the previous lessons and relates it to the phases of the model development process: the conceptual phase, the technical phase and the dissemination phase. This tutorial approaches more theoretical aspects in a practical environment. For the conceptual phase, particular focus is directed at discussing what kind of research questions are suitable for modelling and how to pick the right modelling technique. The importance of properly conceptualizing a model before starting the technical part of model development is highlighted as it is a common issue for less experienced modellers. The experiment design phase, including parametrisation, validation and the analysis and interpretation of the models is explained at length since it often proves challenging for students. On a practical side, the BehaviorSpace – NetLogo’s experiment environment – is explained to enable the learners to batch run their models. Finally, the dissemination phase of model development is also included to ensure that the standards in model publication are widely known, and the learners explore the fundamental importance of replication for advancing scientific knowledge.

3.2 Tutorials and workshops

The tutorials were tested during various workshops and updated afterwards. As mentioned earlier, the participants of these workshops were asked to fill in a questionnaire both before and after the workshop.

3.2.1 Before the workshops

A large proportion of the participants in the workshops gave us information using the survey before the workshops (172 of 368 participants). The respondents came from at least 40 countries that represented almost all continents (see Figure 2). There was a clear skew towards participants from Europe. The number of female respondents slightly outnumbered the male ones, and a small group did not share their gender, while two identified as non-binary (see Figure 3). The ages of the respondents ranged from below 20 to over 70, although the majority fell between 20 and 40 years of age, indicating that the audience was predominantly composed of early career archaeologists. It seems that the different events also had a slightly different distribution of both gender and age. For example, more older people attended the workshop at the CAA conference in April 2023, and more men were present during the workshop at the Reuvensdagen in November 2023. This might be due to the differences of audiences at the conferences. For the Reuvensdagen the number of participants of the workshop was relatively low and this variance might be due to chance.

Figure 2

Figure 3

Most of the respondents reported having a level of computer literacy already including particular computational skills common in archaeology, for example working with word processors, GIS and spreadsheets (see Figure 4), and 111 of the respondents had some prior knowledge of ABM, while 59 were completely new to the subject. The majority (158) had never applied ABM to their research before participating in the workshops with only 14 participants who had experience of developing archaeological ABM. It is interesting to note that many respondents (75) could mention at least one type of software to be used for ABM (see Figure 5).

Figure 4

Figure 5

As shown above, the respondents had some knowledge on ABM in general, but did not know how to apply it or had never applied it before. The respondents were also asked how they rated the available theory on ABM (Figure 6). While 64 of them had no opinion on the subject, a proportion of the respondents (28) answered that they rated the available theory on ABM as limited and only a small group (39) as sufficient or better. This clearly shows the need for more and better educational material.

Figure 6

3.2.2 During the workshops

During the workshops the participants worked in a self-paced manner, often on their own, but sometimes working together and discussing the tutorials (Figure 7). Some participants engaged in discussion with the teachers to learn how they could apply ABM in their research, or discussed possibilities for the application of ABM in general. Individual progress varied depending on the skills and preferences of the participants. While bugs were still a problem in earlier workshops, participant-teacher interaction was more focused on content-related learning problems in later instalments.

Figure 7

3.2.3 After the workshop

A similar proportion of the participants in the workshops gave us feedback after the workshops (171 of 368 participants). The respondents were positive on the workshop in general(see Figure 8).

Figure 8

Most respondents (153) were enthusiastic about the teaching materials, with the majority rating them as excellent or good. The teachers were rated even better than the teaching materials, with most respondents (165) giving them an excellent score (see Figure 9).

Figure 9

Two more open questions were asked to the respondents. The first was aimed at learning what aspects the participants liked in the tutorials (What aspects did you like most?), and the other was aimed at getting feedback to improve the tutorials (What would you like to see changed or what could be better/different?).

The respondents generally liked the interactive step-by-step way of going through the tutorials, but also valued the interaction with the teachers strongly. The easy and intuitive introduction to ABM and NetLogo were also mentioned quite often. This was also observed during the workshops as most participants were working at their own pace and did not need much help. From a didactic point of view, the self-guided and self-paced way of going through the tutorials is very efficient for teachers and limits the frustration of students that may otherwise go too fast or fall behind the class when the tempo is dictated by the teacher.

We also received feedback on possible improvements. During the first workshop at the CAA conference in Amsterdam in April 2023, we did the first test of the tutorials and we also instructed the participants that this was work in progress. For this and the following few workshops we mostly received feedback about bugs causing the tutorial to crash or creating unexpected behaviour. The number of references to bugs was reduced to zero by the end of the project and participants started providing other kinds of feedback, for example the wish to see more examples or expanding the tutorials further. Some wanted to introduce more cooperation between participants, which we did not consider primarily because the tutorials are aimed at individual work. A more cooperative aspect of the education materials is possible, even if harder for online events, and we hope that a wider classroom use of the ABMA materials will deliver solutions. It was also interesting to note that over time fewer people could see additional room for improvement indicating that the level, content and format of the tutorials met their needs.

We also asked the participants about their intentions in terms of ABM and their professional life. The majority of the respondents, 88%, wanted to apply ABM in the future in their research. Some of the respondents were already using ABM in their work. A large group (around 20-30%) was not sure yet how to apply ABM and wanted to read more on the subject or play a bit with the possibilities. Many respondents also shared the context to which they were thinking to apply ABM, mentioning topics such as movement of people or goods over land or water, sometimes in relation to trade or other distribution mechanisms. Others thought of demography, social networks, migration or settlement distribution patterns. The natural environment and the interaction with humans in the past were also mentioned frequently, often in relation to GIS and the question of how complementary these two computational techniques are. The archaeological periods that the participants were interested in were very diverse, ranging from the Palaeolithic to the Medieval period.

3.3 Integration with Learning and Skills Tracking Frameworks

Beyond the creation of the OERs, the project also worked on aligning the learning outcomes of the OERs with the Digital Competence Framework for Citizens and the Digital Archaeological Skills Passport.

38 Analytical Skills

    38.1 Modelling

        38.1.1 Computational Modelling

            38.1.1.1 Agent Based Modelling

                38.1.1.1.1 Softwaremastery

                    38.1.1.1.1.1 Creation of models and understanding of key components

                    38.1.1.1.1.2 Integration of other fields - GIS, network science

                38.1.1.1.2 Analysis and interpretation of results

        38.1.2 Mathematical Modelling

etc.

4.1 Iterative creation of tutorials

The chosen workflow for developing the tutorials, namely testing the tutorials ourselves and then testing them during workshops and conferences, proved very valuable in many ways. First of all, it had a very practical use; new users found bugs that we as a team missed or did not encounter. Participants used different operating systems, different web browsers and different settings on their computers, leading to other uses and other bugs. In addition, different people interact differently with software on their computers. These behavioural differences between users helped us to understand how people would use our tutorials and enabled improvements. Thirdly, our presence at conferences and the organization of workshops improved dissemination of the tutorials. A fourth aspect is related, since the high number of people interested in these workshops has shown the need and demand for educational materials for ABM and to improve computational skills.

4.2 Computational skills in archaeology

Based on the high demand for our online and offline workshops, it is possible to conclude that “resistance to digital tools” (Morgan 2022: 216) might be dwindling. The high demand was striking in Amsterdam during CAA 2023. The large number of registered participants forced us to expand the workshop and to split the group in two different rooms to account for all people. In addition, the number of registrations for the online workshop was far exceeding our expectations. While not everyone showed up, this does show that the interest and demand are high. During the various workshops, we encountered many people wanting to apply ABM to their own research, but lacking the skills or tools. The high demand for our tutorials and workshops shows that there is an increasing group of archaeologists willing to learn about new tools. At the same time, the tutorials help to move ABM away from the fringes of archaeology by offering an easy access to ABM and programming. Of course, some participants who found the workshops interesting may still not pursue ABM further. However, while they may not end up applying ABM in their own research, they are now more aware of the possibilities of ABM and digital archaeology more generally.

Teaching digital archaeology is often contingent on the skills and interest of the teaching staff of a given higher education institute, although some institutes are incorporating digital aspects of archaeology in their regular curriculum (Lambers 2024; Visser, Zijverden and Alders 2016). The number of specialists in ABM is limited worldwide and the OER’s created in this project offers archaeologists an easy introduction into ABM, modelling and computational thinking. However, we encourage people to discuss their ideas with other people in the field, for example during CAA conferences (https://caa-international.org/). We have seen during the workshops that discussion can be an important aspect of model building, one which is challenging to incorporate in self-paced learning.