Involving Users in Infrastructure Development: Methodological Reflections From the Research Pilot Projects Using the CLARIAH Media Suite

The project Images for the Future (2007-2014) provided funding for the massive digitisation of audiovisual heritage in The Netherlands.¹ More than 200,000 hours of film, video and audio have been digitised. In addition, approximately 38,000 hours of digital-born material are ingested directly into the archives of Netherlands Institute for Sound and Vision on an annual basis.² The wealth of sources that have become digitally available, and the emergence of new tools to analyse those collections at scale, have fuelled expectations on methodological innovations that can lead to new breakthroughs in media scholarship.³ At the same time, several humanities scholars have pointed out that such innovations require careful analysis and consideration of concrete user needs. In 2012, Joris van Zundert argued that large-scale, generic infrastructure solutions might in fact hinder such innovations.⁴ Instead, he pleaded for smaller-scale solutions for research with digital sources and methods focused on user needs of scholars and knowledge exchange.⁵ Relatedly, Sabine Lenk highlighted the importance of collaboration between the owners of digital collections and the scholars using them.⁶ Responding to the large-scale digitisation of the Images for the Future project, she emphasised that scholars can bring their knowledge and expertise and as such may meaningfully contribute to the accessibility and usability of the digital sources.

We followed this plea for involving users and collection owners in infrastructure development when building the Media Suite⁷ in the context of the CLARIAH project⁸, that aims to develop the Dutch national infrastructure for digital humanities research. The Media Suite provides a virtual research environment for accessing and researching audiovisual sources and related contextual sources. In 2017-18, the development of the Media Suite included its testing and use in six research pilot projects that were selected in an open competition.⁹ The aim of the pilots was to involve the researchers during the period of one year in the co-development of the infrastructure together with the information technology experts and the other scholars in the CLARIAH Media Studies team.¹⁰ To implement this co-development process, we asked the research pilot project teams to write so-called demonstration scenarios, in which they outline all the steps in a research process focusing on one specific theme. These scenarios, then, were the sources that provided concrete requirements for the development of the Media Suite. The process of going back and forth between the description of a concrete research process and the data and tools made available via the Media Suite invited methodological reflection. Two of the six pilot project teams have published their methodological reflections during their projects.¹¹ In this dossier in TMG Journal for Media History, the researchers of the four other projects share their thoughts on the methodological dimensions of doing media scholarly research with the CLARIAH Media Suite sources and tools.

The CLARIAH Media Suite was officially launched in April 2019. It is a virtual research environment that provides access to audiovisual collections and the tools to search, visualise, enrich, and analyse them. The Media Suite provides online access to, among others, the entire audiovisual collection of The Netherlands Institute for Sound and Vision¹², the Eye Film Museum Jean Desmet collection (included by UNESCO in their Memory of the World register)¹³ and the oral history interview collections maintained by DANS¹⁴. Previously, these collections could only be accessed separately or on location because of copyright and privacy issues. The Media Suite works as a data aggregator of those collections, and as a research environment in which scholars can access both the metadata and the media content itself. Researchers can log in to this environment using their institutional affiliation credentials, and then search, explore, browse, store their selections, analyse and annotate, and visualise the outcomes of their analysis in a personal workspace.

A barrier to tailor infrastructures to scholarly work is that, traditionally, the design of information systems was mostly focused on the technical aspects, incorporating user input only at the very end of the chain. As Claire Warwick recounts, it was generally considered as simply not useful to ask users about what they needed ‘because they would not know’ and to ask them to provide feedback ‘because they would not care’.¹⁵ The technologists were the ones who knew best what to do and how they would work. A more user-centred design methodology started to gain ground already from the mid-1980s onwards and moved the attention to the users’ needs as the starting point for designing information systems.¹⁶ Franciska De Jong et al. explain as follows: ‘The typical ICT cycle starts with a user study and the analysis of the requirements that need to be fulfilled in a scenario of use or for a specific task for which users seek technology support.’¹⁷ With the research pilots, we followed this approach to involve users and implemented a two-staged format for co-development.

First, the pilot scholars were asked to write a so-called ‘demonstration scenario’ document, which is a description of the context and processes of the users’ goals and research phases, in a specific scenario of use for an infrastructure which is ideal for the use case but is not in place yet. They follow the typical steps of research (from exploration and selection over contextualisation and analysis to publication) and describe their actions. In order to be specific, they refer to other existing facilities to provide examples of the envisioned tools and functionalities.¹⁸ Through those demonstration scenarios, the information and system experts can understand how the system could support the scholars’ goals and can draft a list of requirements. For the development of the Media Suite, these requirements were documented in the format of a user story (see Fig. 1), allowing the developers to understand the goals and context of the request.¹⁹ The requirements were subsequently ranked according to priority and grouped together in terms of technical dependencies, after which the developers could start implementing them according to priority. The pilot scholars, then, would make use of the system, providing feedback after every release that was incorporated to the iterative development and testing of the Media Suite.

Figure 1

Secondly, while doing their research, the research pilot scholars were also invited to reflect upon the affordances of the digital data and tools in the Media Suite in-the-making. The idea was to encourage a ‘tool criticism’ approach’.²⁰ That is, to invite scholars to be explicit about how they use a specific tool in their research process and evaluate what it brings and where its limitations lie. Reflection is typically done at the very end of a project, by revisiting the research process, the results and research notes. In the case of the research pilots, they could not only make use of their research notes, but also compare the results with their initial demonstration scenarios and reflect on what they wanted to do at the start of the project, to the results and the obstacles encountered along the way. This second stage generated an additional set of requirements for adjusting the tools and interface for later versions of the Media Suite, which were intended to be implemented in the years after the pilot projects ended.

It is this second, reflective phase that forms the basis of the contributions in this dossier, which present and discuss the methodological aspects or questions that came to the fore when using the Media Suite. The pilot scholars meticulously show how they worked with the limitations and challenges as imposed by using a large-scale research infrastructure in development. In addition, they provide insight into the methods they designed for working with the then available functionalities of the Media Suite and suggest concrete routes for improvement. The contributions shed a unique light on the research practice in the digital humanities by providing real life and concrete examples of use of a specific environment for researching audiovisual data and contextual sources. By doing so, the dossier also responds to the recommendation of Fred Gibbs and Trevor Owens to incorporate concrete methodological examples to prospective users of digital tools.²¹

The four pilot scholars that contributed to this dossier all hail from different backgrounds and taut a variety of collection-expertise. Historian Susan Hogervorst, principal investigator of the CrossEWT project, investigated the topics discussed in oral history interviews conducted with World War II veterans. With her project, she also contributed to gain understanding on how the Media Suite could support historians and especially historians working with oral history collections. Her project is centred around the accounts of eyewitnesses about World War II in the Netherlands since 1945, using a cross-media approach to working with newspapers, television documentaries, and oral history interviews simultaneously. In her reflection, she focuses on the difficulties for building her corpus during the initial phase in which the CKAN collection registry²² and the Media Suite were built, and how she found workarounds and solutions to achieve her research goals. She ends her chapter with a reflection on the benefits of video annotation tools for oral history research and the wider challenges of using an infrastructure built for audiovisual collections for historical research.

Film scholar Christian Olesen was the principal investigator of the MIMEHIST project, which facilitated the integration of the UNESCO world heritage Jean Desmet collection into the Media Suite and simultaneously investigated the connections between films, posters, and Jean Desmet’s business documents. In his contribution to this dossier, Olesen focuses on the methodological background and rationale for using and providing input for tailoring the existing annotation tool provided by the Media Suite to facilitate mixed media film analyses.²³ Enhancing the annotation tool was one of the aims in tackling the problem of establishing and exploring meaningful links between mixed media collections such as the Jean Desmet’s collection.

Media scholars Sabrina Sauer and Berber Hagedoorn, principal researchers of the NarDis project, investigate in their contribution how to trace back the creation of stories or narratives of disruptive events by television, newspapers and radio reports and how these are instilled with specific cultural-political meanings. They look at how researchers tell stories about two specific disruptive events, namely the assassination of Dutch politician Pim Fortuyn and the North Sea flood of 1953, based on nine user sessions with the Media Suite. They elaborately discuss the steps and challenges that their participants encountered when using the former DIVE+ recipe²⁴ (currently named Explore Tool) and an early version of the Comparative Search recipe (currently named Compare Tool) in the Media Suite. In their final section, they outline pathways for future functionalities, and make a strong plea for the incorporation of more collections and especially collections in linked open data format that can be used across tools.

Figure 2

Lastly, journalism scholars Mark Vallinga, Frank Harbers and Marcel Broersma, the research team of the ReSpoNs project, investigated how sports journalism in newspapers has changed in response to the rise of television between 1959 and 1989. They show how they worked with the first versions of the Compare tool in the Media Suite and with Delpher²⁵ as one of their main collections were newspapers. In their article, they compare challenges and pitfalls by juxtaposing the textual newspaper collection and the audiovisual television collection. They make recommendations on how an improved Compare Tool can do better justice to the specificities of these collections, and how this in the end could lead to cross-media research within one infrastructure. By the time of this publication, the CLARIAH Media Suite is fully functional and aimed to be sustained at least for the forthcoming decade. The pilot projects have been conducted with Media Suite version 2 and 3, whereas presently version 6.1 is available.²⁶ The majority of the needs envisioned by the research pilot scholars in their contributions, therefore, are already implemented in the new versions of the Media Suite. This included, for example, the option to do comparative research across the Radio and Television collections and newspaper collections, which is now possible via the Compare tool, and the integration of ASR (automatic speech recognition) transcriptions for a large part of the Radio and Television collection, which now can be searched in an equivalent of full text search for the newspapers in Delpher. Other wishes will be possible with the implementation of Jupyter Notebooks that will enable direct work with the data from these collections.²⁷ We invite all readers of TMG Journal for Media History to explore the methodological reflections in this dossier and to visit the Media Suite via mediasuite.clariah.nl to dive into the various audiovisual collections and digital tools. Ideally, this will result in new use cases and user requirements that will inform the future improvement of the Media Suite as a core infrastructural facility for researchers studying audiovisual collections and related materials.