From Stock Shots to Ghost Data: Tracking Audiovisual Archives about the European Union

1 Introduction

There is a growing presence of audiovisual archive material in media spaces¹. These stock shots circulate and cross geographical and institutional frontiers, and thus contribute to defining our shared memories. The image we have of the past is determined by these visual forms shared through the media. The process of reuse and sharing of audiovisual archives is a form of mediatization of memory². Media industries thus play an active role in the construction of collective memories, as do the archives from which they retrieve audiovisual material. If the mediatization of archives is a subject of growing interest in the field of public history and media studies, the process of circulation itself is still quite a rare object of scientific study, despite that circulation has always been, and is more and more evidently, a fundamental factor in the life of images and videos. By crossing different borders, content gets contextualised through different devices, situations, and environments. Images pass through different enunciation regimes³, genres, and social and political situations, affecting the meaning and thus the social impact of documents.

The specific case that we are studying is the reappropriation of footage that had been shot for other original purposes. These images are interesting because they are authentifying documents⁴: audiovisual archives show themselves as a trace of the past, the recording of a real event achieved by the visualisation of objects or persons, and this contributes to its status as a proof of realism for the viewer. If any archival item is an authentifier in this sense, audiovisual ones also possess a temporal dimension (length), which gives special interest to these documents. The existence of the European Union begins at a certain known date because a treaty has been signed that day, and to show this footage again and again within news programs means trying to prove (again and again) the historical reality of the event. And thus, to make it real to the eyes of the viewer, in a way that is only possible by means of direct or indirect perception, instead of by simple wording and phrasing. Within the theoretical framework of the project CROBORA, we collected images related to what Andrew Hoskins calls flashbulb memories⁵. This concept is used to refer to those memories that are experienced by individuals as images that vividly recall an event of the past. The repeated occurrence of some images in the media plays a similar role, creating and keeping alive, in an externalised form, some “memories” sharable within a community⁶.

This paper is based on the research project Crossing Borders Archives (CROBORA), founded by the French National Research Agency, 2021-2024, led by the Sic.Lab at Université Côte d’Azur in partnership with the French National Audiovisual Institute (INA), RAI (RadioTelevisione Italiana), Mediaset, Luxembourg University, the Catholic University of the Sacred Heart in Milan, the Universities of Lille, Paris Nanterre, and Paris Sorbonne. The general objective of the project is the collection and analysis of the reuses of broadcast archives within television programs and web videos. The project deals with images linked to the construction of the European Union. CROBORA will build a cartography of the visual memory of the European historical constitution through the reuses of audiovisual archives.

The objective is to understand the dynamic dimension of collective memory. The past changes over time, and CROBORA aims to understand how the media determine the dynamism of visual memories. The memory of an event may be carried by audiovisual archives shared in the media, the hypothesis being that when these images are reused and recontextualized in different environments, that also changes the memories they are supposed to represent. The handshake of Kohl and Mitterand in 1983, for example, was initially used in French TV to represent the Franco-German reconciliation, but after the Maastricht Treaty of 1992, it is increasingly used as a symbol of European construction. It has become polysemous because of these reuses. In this way, not only the meaning of the image is determined by these mediatizations, the visual memory of the community is also influenced by them. The project thus aims at studying repetitions of audiovisual archives to understand how different media environments affect the collective visual memory.

To undertake the data collection and constitute the field, two perspectives could be adopted by the researchers. The first, the simplest and often used in the human and social sciences, was to adopt a deductive approach. This means identifying a certain number of case studies, and then carrying out targeted queries in databases aimed at finding these same images. This approach is used by large-scale research projects, such as the HIVI project, led by Valérie Schafer and Frédéric Clavert at the University of Luxembourg, which aims to collect significant memes to understand the phenomenon of virality⁷. The collection carried out within the Artlas project, led by Béatrice Joyeux Prunel at the University of Geneva, puts into practice a real digital Atlas Mnémosyné⁸, dealing with databases of artworks around the globe. The objective is to make an epidemiology of images: to understand how they circulate and the exchanges necessary to their dynamism⁹. The CADEAH project offers such a perspective on European audiovisual corpora, those of EUscreen and Europeana. Fingerprint software is thus used to find occurrences in each corpus. These perspectives are generally deductive in the sense that they start from a corpus of given images and then find their occurrences within a field. This makes it possible to understand the logics that determine the circulation of certain images but does not make it possible to have an exploratory approach meant to find which are the redundant images linked to a given phenomenon. We therefore proceeded with a different approach which is data-driven: within a given perimeter, we searched for all the reused stock shots¹⁰ related to the chosen subject to answer the question without defining them a priori. Then, only after having identified a sample of repeating elements, we addressed the question of understanding how they circulate. This means that instead of starting from a set of given videos to look for within a field, we wanted to understand which videos are repeated in the first place.

A large part of our data comes from two national archives (the legal deposit of the French television kept at the INA, the audiovisual archives of the RAI in Italy) and a private deposit (the archives of Mediaset). These huge archives present the 24-hour feeds of national television channels and are protected by specific national rights. To find stock shots, we have thus used the documentation of audiovisual archives. These institutions perform a great work of indexing that allows us to trust the textualizations to find, if not all the stock shots, at least the most obvious ones to the documentalists. This means that when stock shots are reused in a television program, the presence of images is documented by RAI or INA and manifests itself in the metadata. However, for all the archival deposits documenting TV broadcasting, the basic unit of document is supposed to be a subject (a piece of news report for example). So, via textual documentation, we could only find the whole video containing certain stock shot(s) in it (a news report on TF1 at 8pm for example), not the stock shot which is still embedded in its initial context. From the first collection based on textual search in databases, we then engaged in a second collection to pick up only redundant stock shots that interest us.

A similar approach could be largely found in the studies on text tracking, with certain important efforts in algorithms realised to identify recurring texts, such as the Viral Texts project, which aims to detect, substantial repeated passages of tens of thousands of words embedded within unrelated newspaper issues¹¹; as well as other projects which are interested in finding repeated “memes” of only a few words.¹² In the visual domain, data-driven studies using automatic approaches have also augmented in recent years. But when it comes to track identical stock shots in large collections of audiovisual archives, the problem seems trickier.

2 Ghost Data

Studying the circulation of archive material means that we are interested in the process of repetition: the redundancy of the same items found at different times in different media. The main object of our research is more specifically the recurrence of the same audiovisual sequence (thus a stock shot) within a corpus aimed to be representative of an ample international media segment. To understand the circulation of a stock shot, we aim to find all its occurrences within a given period. It was thus necessary to proceed to an analysis of a given field to understand what is repeated and how the repetition is affected by different logics of refashioning and remediation of images.

However, we face the fact that each occurrence of stock shot, visually repeated within different video programs, appears as a different document in the databases (first issue). Furthermore, stock shots come as fragments of longer video sources that are not necessarily present in our corpus in their entirety (second issue). These issues show that the core object of our research is in some sense a ghost, for two reasons: first, like an emanation coming from the past, stock shots belong to original footage which is often inaccessible; second, like an ethereal being, the stock shot, repeated through different forms and visual appearances, is there somewhere in our corpus but we cannot really know beforehand where exactly.

We thus call ghost data those items that are materially present in a corpus, but for which we lack sufficient documentary evidence to identify them. Given that a document requires a treatment process to become such¹³, we can assume that an item lacking those elements allowing it to be precisely identified is somehow absent from the corpus despite its fragments and traces belonging to it empirically. This makes it a ghost, something that we could be willing to say that it exists and that it does not exist at the same time. Ghost data is thus an element whose existence is latent in a corpus, as it can only be revealed indirectly by other empirical traces or interpretants¹⁴. Excerpts, traces, or fragments of this item do appear, allowing us to get a glimpse of the data they point to. We are interested in stock shots whose occurrences are present repeatedly in a corpus, while this repetition is neither documented nor indexed as such. Certain clues lead us to see that we are in fact in front of repetitions, however, the fact that it is one and the same item repeating itself is not represented to the machine. We emphasise the importance of understanding this notion as an essentially empirical and operational question, and not as a purely theoretical idea. Ghost data is not some general information hidden in a corpus, which has been somewhat of a trivial notion for data analysis at least since Tukey¹⁵, but a factual element that requires some analytical and computational effort to be individually identified.

The concept of ghost data is useful to understand how to manage some (visual) items within a dataset. The issue is how to work with stock footage whose occurrences are scattered in various forms. Our solution was to collect full videos (news items) and their metadata, then we took screenshots for each reused stock video sequence, and we annotated these screenshots. Despite this very laborious and almost totally manual process of data collection and enrichment (which involved one PhD researcher, one Postdoc researcher, and 8 interns), what we finally get is different images that are not yet indexed as individual occurrences of a smaller number of repeated units or types. For instance, we can have 70 documents where the signature of the Treaty of Rome is shown by using stock shots that may be the same, and probably come from the footage shot in Rome by RAI in 1957. We may thus have information telling us that the same footage of the Treaty of Rome has been cited 70 times. However, if we don’t watch the videos manually, we cannot know whether it is the same footage (shot from the same angle, etc.) or not. So, to track down the path of each repeated stock shot, we should watch them manually, which would be possible only on a small scale. Some techniques permitting distant viewing¹⁶ are thus necessary to move on a larger scale. Instrumentalizing the analysis of visual documents, the project thus implemented a quantitative semiotics approach¹⁷.

Moreover, even if we identified several occurrences as repetitions of a single type (a solution to the first issue), we would still not have the video source, which means the complete length and original video from which these shots have been extracted. In the case of our example about the Treaty of Rome, the video source is kept by RAI and Istituto Luce (the second issue). What we do have are different parts of the video source, different timecodes, and even different edited versions. Ghost data are thus those items that need to be fully identified and somehow indirectly recovered from a dataset. We are now going to detail how the stock shots collected for the CROBORA project allow us to retrieve ghost data through their collection and documentation. In the following sections, we intend to describe the way in which we approached the corpus construction and the detection of ghost data.

3 Corpus Construction

3.1 Data Sources and Search Scope

The phenomenon that the presence of audiovisual archive material in media spaces is growing can be clearly observed in our research scope. The two graphs below show a tendency of a growth of stock shots both on France and Italian territory, precisely from 2012 for France, and from 2016 for Italy. This observation proves a necessity to take a closer look at these specific archives. Based on this ‘archival rise’, we have chosen to collect the stock shots from different sources.

Figure 1

Figure 2

Our data collection is based on several heterogeneous data sources. First, our research field covers television in France and in Italy, which means the implication of multiple repositories of different natures: the French Legal Deposit of Web and Television, held by INA; the archives of the Italian public service of television, at RAI; and those of the main italian private broadcaster, Mediaset. We also collected data from a selected number of social media accounts on the French and Italian web.

A first problem comes from the heterogeneity of data sources, which leads to a heterogeneity of metadata structures. On the one hand, RAI and INA archives, being conceived as public heritage institutions, present highly detailed metadata for the programs they gather; however their style, language and criteria are different: for example, while RAI annotates both the journalistic discourse in voice over and the visual content in a program, INA focuses more on the discourse made by journalists and anchors, and tends to categorise the footages instead of describing their visual content (“images d’archives”, “images factuelles”, “extraits”, etc.). On the other hand, Mediaset stocks documents with different aims, not being interested, as a private player, to document broadcast programs for the aim of preservation. Mediaset archives serve a dual purpose: to collect everything that has been shot by the broadcaster’s personnel and to retain what was broadcast for potential future reuse. Furthermore, social media accounts are documented in a non-systematic way, starting from the descriptions used by each account. While these original metadata may be interesting for textual analysis, enrichment by annotation had to be conceived and executed to harmonise them.

For the French and Italian public television archives, we limited ourselves to the genre of news broadcasting, to the period from 2001 to 2021, and only to some major national channels: TF1, France 2, and ARTE on the French side, Rai 1, Rai 2, and Canale 5 on the Italian one. The search queries were based on the words “archiv*/repertorio” and “europ*”: in this way we aimed to collect most stock images concerning the European Union. To this day, we successfully gathered a total of 22,520 instances of stock footage from television archives. As for the Web, there exist two approaches for data collection: firstly, the legal deposit of the archived Web curated by INA constitutes the first data source; secondly, we also collected straight from the live Web¹⁸. Therefore, two methodological approaches were involved: textual search in the INA database and search engines of the live web. In this way, we had to rely on the previous enrichment of visual data by humans who documented this data, either by archivists or online professionals. However, the problem with the web lies in the fact that with a large quantity of videos, the metadata is much less structured than that of television. In the absence of structured metadata, it is not possible to filter the data and then visualize all these videos through human efforts, as we did for the television data.

As for the live Web, our focus spanned both Italian and French territories from 2001 to 2021. We selected various EU official accounts such as the European Commission, European Parliament, Palazzo Chigi, Farnesina, Commissione Europea in Italia, FSE Regione Basilicata, Movimento Federalista Europeo, Ministère de l’Europe et des Affaires étrangères, among others; Additionally, we incorporated media accounts like Euronews, Istituto Luce, Treccani Scuola, Zanichelli Editore, Eunews, Istituto Luigi Sturzo, and more from platforms such as YouTube, Dailymotion, Twitter, and Facebook. These were chosen based on a series of general keyword searches including “Europe”, “Archive”, “History”, as well as more specific searches like “Treaty of Rome”, “Maastricht Treaty”, “Berlin Wall”, etc. Additionally, certain media websites, such as Arte.tv, were also included in our sources. Generally, our data primarily originated from heritage institutions, media sources, and EU institutions. We managed to accumulate 5,393 instances of stock shots from the live Web. It’s worth noting that our data collection process heavily depended on pre-existing metadata and recommendation algorithms from video platforms, which means the exhaustiveness of our dataset might not be fully ensured given the somewhat “random” nature of our keyword research. Additionally, it’s important to acknowledge that any online content that has been removed from the original platforms or websites leaves no trace on the live Web. This is precisely why we also resorted to the French legal deposit of the Web, which boasts a collection of 16,069 media websites that specialise in audiovisual content. From this resource, we leveraged an expansive, unfiltered corpus consisting of 56,000 videos. These videos, published on platforms such as YouTube, Dailymotion, Twitter and Facebook between 2001 and 2021, were selected based on their metadata containing the keyword “Europ*”. This body of videos was curated and imported by INA archivists. A detailed automatic exploration of the corpus will be provided in part 6 of this article.

We chose from 2001 to 2021 as our limited period to demarcate a limited set of news reports for our data collection. What’s more, this period is also considered as rich in the construction of Europe, a time range that we try to divide in two phases: from 2001 to the late 2000s, twenty years of institutional and diplomatic work between the Maastricht Treaty and Treaty of Lisbon, when it comes to the choices of the euro and an independent European central bank, the thunderclap of the rejection of the European Constitution and the adoption of the Lisbon Treaty without fanfare, as well as EU civil-military engagement in the Western Balkans, etc.; secondly, from 2010 to 2021, where we have seen some unprecedented challenges, including the European debt crisis, the EU facing the challenge of migratory attraction, the rise of illiberalism which breathed new life into Euroscepticism, Brexit, and then, of course, the COVID-19 pandemic.

In our constructed corpus, we have observed two slight peaks separately in the year of 2005 and 2011, the reasons for which lie in the victory for the “No” campaign during the referendum on the Treaty establishing a Constitution for Europe in 2005, and a stock market storm occurring in 2011, partly due to the Greek public debt crisis; meanwhile, a surge from 2014 to 2019 is detected. Apart from the 2015 European migrant crisis, another hypothesis is that the archivists in INA and RAI for example tend to deal with more recent news reports. As our initial research is based on the metadata accompanying the audiovisual archives, a richer documentation work done by archivists would probably help us find more results than earlier years in the database. This hypothesis related to metadata could be partly proved by the drop after 2020 - the high volume of news about the pandemic has a great impact on archivists’ workload when it comes to document indexing. The reports which should be initially indexed as « archiv* » and « europ* » could be left behind during this special time. While the database is large, there are significant gaps in its holdings and problems with its data that influence our results.

Figure 3

3.2 Data Types

Our corpus is made of different kinds of data and accompanying metadata, listed below.

3.2.1 Videos

Audiovisual artifacts are the research focus of our project. Thanks to our partnership with INA, Rai Teche, and Mediaset, we can obtain all the videos in our perimeter archived by these repositories. We also tried to download all the relevant videos from the live Web. However, all these videos are non-segmented, which means that the specific stock shots that interest us are embedded in their original context. Therefore, one of our main research challenges is to extract the stock shots embedded in these videos.

3.2.2 Key Frame of Each Stock Shot

The main solution we adopted to extract stock shots has been to manually identify the sequence in the video and to capture a screenshot for each one. In our approach, the first frame per stock shot is used as a keyframe. These screenshots act as signs (indexes, to be precise) of the video sequence itself. The screenshots are a manipulable substitute by which we operationalize the task of working with video sequences¹⁹. As Lev Manovich also writes, to visualise a video collection, it is usually more convenient to select some frames that capture the properties and the patterns of each video and work with them²⁰. The technique of key frame selection to represent a video has been largely developed in the computational domain²¹. The proposed technique is composed of three steps: shot boundaries detection, shot selection, and key frame extraction within the selected shot. For example, the “Distant Viewing Toolkit” (DVT), a Python toolkit for computational analysis of visual culture that addresses the challenge of working with moving images through automatic analysis such as narrative arcs detection²². For the CROBORA project, the challenge is that it is not the whole content of videos that attracts our attention, but the very fragmented stock shots hidden in each video. We considered in the first place to automatically extract key frames for each shot in our initial corpus, whether it is a stock shot or not, and then, using algorithms to cluster all those key frames according to their visual characteristics, hoping to be able to retrieve identical images from the result. For example, we applied PixPlot²³ on our corpus. It turns out that some general trends could indeed be detected by the software, such as clusters of all European flags, documents, or meetings of European leaders, etc. Nevertheless, unique phenomena could be hidden behind these big trends. That is to say, this kind of software probably mixes similar images together with identical ones. Due to the high volume of images that we have, it is nearly impossible to detect identical images from those big chunks created by computer. Therefore, we consider manual qualitative work necessary, in a way that researchers must view each video to dig up those hidden sequences, to collect and only collect stock shots. Here, each key frame is manually captured with the progress of our data collection.

Figure 4

3.2.3 Original Metadata of Videos

Original metadata are the text accompanying the videos that we collected, such as documentary records produced by documentalists of INA, or video descriptions written by official accounts on Youtube. It was during the process of data collection that we first realised how important and possibly influential metadata were for the construction of our corpus and we decided that further studies on them could be proven interesting. In fact, the process by which the metadata are originally entered by the institutions who manage the collections imposes cultural categories on data themselves. Furthermore, as an important form of documentation of digital/digitised artifacts, these metadata constitute elements for textual analysis with the aim of understanding the context and meaning of the audiovisual objects that we collected.

3.2.4 CROBORA Metadata of Video Sequences

The fine-grained annotation for specific sequences is a very valuable supplement to previous institutional metadata. CROBORA metadata are new descriptors added to each stock shot by the project’s participants. If for the original metadata produced by institutions, the focus stays on the resume of the whole video content (in which the stock shots that we are looking for are embedded), our purpose is instead to establish a form of re-coding that focuses only on the stockshots. We have chosen four aspects to describe each sequence: personality, event, place, and illustration. English is used for both French and Italian metadata. In addition, each sequence has been attributed a unique identifier (UID). It is important to remark that qualitative annotation of new metadata considers the full news items in which the shot appears, so to correctly retrieve its meaning in context. Automatic labelling is a well-developed domain in visual analysis, with many tools like DVT capable of automatically producing metadata summarising the content (people/actors, dialogue, scenes, objects) and style (camera angle, lighting, framing, sound) of the images. For example, in a study carried out in large historical collections of the German Broadcasting Archive, an approach of content-based video retrieval has been adopted²⁴. The aim is to automatically assign semantic tags to video shots. This method could be very interesting for our annotation process. Nevertheless, we have not found a software which could satisfy our specific needs. Due to the lack of a thesaurus/prepared lexicon and trained algorithms dedicated to visual representations in a European scope, most of the automatic indexing tools stay at a low-level annotation focused on general and plastic features (people, car, tree, bright, portrait…), while what we need is a form of annotation way more precise and figurative (Angela Merkel, Treaty of Rome, Paris, Border control, etc.). Besides, millions of training images are needed to build a deep CNN model from scratch, which largely surpasses the limits of our project when it comes to expensive and time-consuming costs. Therefore, manual indexing seems the quickest and easiest way to deal with our data from the start. However, a homogenization to standardise human indexing and construct a thesaurus has proved to be necessary in the later phase, which we will not detail in this paper.

Figure 5

4 Typology of Stock Shots

Based on our corpus of stock shots in the news about the EU, we found that their usage doesn’t refer automatically to history as a reminder of the past. On the contrary, it is even rather rare that a stock sequence is reused for a purpose that could be said “historical”. Most of the time, stock sequences are used to illustrate a recent event or even to indicate an event to come, as it can be expected by the prevalent interest of news programs in current affairs. This observation urges us to take a closer look at our corpus and, especially, to try and categorise the collected images.

We identify three types of stock footage in news reports, which we call “illustrative stock shots”, “recent stock shots” and “historical stock shots”. We note that the illustrative ones are the absolute majority over the two other types. Let us focus on this first type. The recycling of illustrative images should be interpreted as a redundancy of a visual form rather than as of specific video content. In other words, it is not the images themselves that are recycled, but the visual types²⁵. For example, the sequence containing a European flag in Brussels filmed in 2007 could be used with the same function as another sequence where a European flag was filmed in Paris in 2022, if they look visually similar to each other for the audience. What is seen is the illustration: an image token like, this is there just as a placeholder for the type. In our corpus, the ones most frequently used represent the European flag, the flags of the core member states of the EU, the headquarters of the European institutions, the hemicycle of the European Parliament and the euro currency. As it becomes evident by reflecting on these examples, their meaning is largely generic: they can be inserted into different contexts to integrate very different discourses.

As for recent shots, these are video sequences used to talk about something that occurred recently, usually less than one year till the present time. Journalists tend to mention these events on account of their “close relationship” with the latest situation. Here is an example: at the end of 2015, to summarise the past year, France 2 broadcast a news report entitled “Année de la rupture” (Year of rupture), by employing some shots on migrants heading towards Europe, followed by sequences about the November 2015 Paris attacks, and then by Marine Le Pen and Nigel Farage meeting with their supporters. This is what we term recent stock footage, where archived images are used to build a connection between events considered closely related. The function of recent shots is to visually build a preferred or model interpretation for the latest event²⁶. It is one of the strategies used by news broadcasters to build a frame around the events²⁷.

Figure 6

Figure 7

Lastly, there are the historical shots, which are found at a significantly lower frequency compared to the other two. They could also be called testimonial shots because their function is to keep traces of what once took place. Historical shots are often endowed with certain visual properties that highlight to the audience the old age of these sequences – such as black and white and poor picture quality. The presence of someone who has already passed away is also a clue to the historicity of the shot. For example, this kind of footage is used to refer to an explosion happening during the Second World War. Historical shots produce in the viewer an effect of reality²⁸, as all their details, not necessarily meaningful in themselves (clothes and uniforms, objects, gestures, etc.), serve as reinforcement to trustworthiness. Hence, the viewer feels almost as an eyewitness: she has in front of her eyes the visual proof of what the news are saying. This lends a stronger persuasive power to the argument, as the points made seem to arise directly from the visual evidence presented in the shot. Unlike the illustrative and the recent stock shots, which are continuously renewed with time, many of the sequences of the historical kind circulating in the media come from a restricted and constant ‘bag of images’. In this case, as opposed to illustrative shots, it is no longer the visual forms that are conveyed, but rather the sequences themselves, the events as specific content, specifically shot at a unique moment in time. The visual tokens work as direct references to the events in their occurrence, almost as proper names do in language²⁹.

Based on our research aims, we’ve chosen to introduce how to trace historical stock shots in this article. This type of footage is extremely interesting to us, as the usage of historical shots is rare but significant at the same time. As an estimation, in our corpus, only approximately one-tenth of the occurrences are historical stock shots. A memorable moment³⁰ must be able to summarise a complicated event in a few seconds. Thus, these images become part of the common memory and heritage, serving as symbolic representations of events crucial to the identity of the European Union. It is not so often that historical events are mentioned in a news program; but when this happens, it seems that it’s always the same images, from the same ‘bag’, that are being used.

Figure 8

5 Tracing Historical Stock Shots

Once we’ve decided which type of shots we want to trace as a priority, the question arises as to what exactly to trace, and how. Historical shots are hidden by a much larger number of illustrative and recent shots and are scattered everywhere in the corpus, and we do not know their entire distribution. For example, during the collection, we detected that some highly similar shots about the signature of the Treaty of Rome have been circulating across time and space. Technically, these occurrences are independent digital objects, as they are the product of independent takes. On the other hand, we want to be able to grasp the idea that they refer to the same event, and roughly in the same way, so that they are variants of the same shot (original or source footage) as we may want to call it. It is not always possible to simply say that the same shots come from the same original footage, and there may be several original video sequences used as interchangeable source material for further reuse. There was a particular event, let’s say the signature of the Treaty of Rome, and different operators caught this event visually, in similar but slightly different ways. Nowadays, these variants may be used interchangeably, all shots referring to the same event with similar formal characteristics (black and white, close-ups on the main politicians, detail shots on the hands, and so on). Therefore, from a semiotic perspective, the fact of whether they all come from the same footage is secondary, while it remains a priority for historical research. How can we aggregate these scattered fragments to piece them together into a whole? Which other historical stock shots should we consider as the most salient in our corpus?

One of the solutions we adopted to deal with this issue lies in the new metadata created by our project members. As we mentioned above, we have annotated every stock shot assigning it to one or more specific media events. Since events aren’t intrinsically embedded in an image, it is clearly the journalistic discourse that makes a shot refer to a certain event: considered from a culturally grounded perspective, reference is realised by semiotic means³¹ . Therefore, we have made recourse to the full videos, offering a context to the shots, including the verbal language used by journalists with the effect of linking the shot to some event. Our manual annotation of the shots therefore required an attentive interpretation, enriching the visual data with dense metadata (we use dense here with reference to Geertz³²), capturing the intention of use as it appears to the eyes of the model viewer. The attribution of events to shots allows us to categorise and aggregate them accordingly. This method works best for historical stock shots, as a historical sequence is steadily related to a specific event. Its repetition over time has already given itself a status that allows it to encompass “the event as a whole,” and the press tends to use it as such. This capability of swiftly summarising events in return restricts the significance of some historical shots. So, there are two different “logics” behind the reuse of stock shots: either a stock shot is so strongly associated with one and only one event that it works as an unambiguous visual metonymy, or on the contrary one shot may point to very different historical events, producing richer connotations. As an example of this latter case, we found in our corpus that the printing of euros could be reused to talk about the European debt crisis, the budget of the EU, Brexit, etc. As an example of an unambiguous interpretation, the sequence of the Schuman Declaration can generalise the creation of the European Coal and Steel Community by itself (as it is not found in association with any other event). Hence, by aggregating historical shots according to the events attributed to them in news reports, we can gain some insight into the construction of historical events about European integration and the logics of their visual representation.

As a result of preliminary analytical work, we have chosen to assign the stock shots to a total of 53 historical events based on CROBORA metadata. Each of them is associated with a varied number of historical stock shots that have been used to visually construct this specific event. The “size” of the events is varied: as an event isn’t an ontologically constant entity³³ , it depends on a framing procedure, whose parameters have been chosen pragmatically to optimise analysis. For example, we could think of Brexit as one main event, or instead as a sum of smaller events (the referendum, the negotiations, the official leave, etc.)

6 Application of a Visual Research Tool to Find Ghost Data

We managed to collect and manually verify and annotate a corpus of 27,000 occurrences of stock shots. We are confident of the coherence of our corpus, that is, that it includes only pertinent stock shots. The question arises as to whether this corpus is also exhaustive, that is, whether it includes all the occurrences of all stock shots used by the media in our perimeter to talk about Europe. As a reminder, we collected this corpus through textual queries prompted to digital interfaces, therefore we had to rely on the precedent enrichment of visual data by archivists. As the question of full exhaustivity risks to remain without a definitive answer, we want to find other ways to prove the representativity of our corpus, and especially to increase its size to include more stocks that our verbal queries might have missed. By doing so, we will also be able to compare our first corpus with the enlarged one, and check whether we observe some systematic bias in our collection that may be interpreted as a sign of global non-representativity.

There are two ways by which we plan to enlarge our corpus. First, by looking for the stock images we already identified as pertinent within the much less structured Web data we obtained. This refers to an additional 56,000 videos about Europe, imported by specialists at INA. In absence of structured metadata, it is not possible to filter the data and then to visualise all these videos by human efforts. It therefore becomes necessary to employ an automatic detector of images, as the one in development at INA under the name Snoop. The second way by which we want to validate the exhaustiveness of our corpus, is by automatically detecting more occurrences of the stock shots we already identified as pertinent in the full corpus, also with the help of Snoop.

Based on a content-based detection technique, Snoop is an interactive visual search engine which is solicited by an initial input of a few images chosen by the user and representing their search intention. Iteratively, similar visual content found by the software is then marked by the user, either positively (those that match her search) or negatively (false positives, retrieved images that do not match with the researcher’ intention). A similarity search query can be very subjective depending on a specific user in a given situation. Snoop then uses this information from the user to strengthen a model of the user’s intention³⁴. By feeding the software with the historical stock shots manually identified by us, we launched Snoop on a corpus of 6,000 news programs plus 56,000 new videos.

By using the software Snoop, we not only intend to extend and validate our corpus, but more specifically to achieve the research of those hidden repetitions that we called ghost data, by: 1. Retrieving as many reuses of one specific video as possible, a step which plays a critical role in the process of drawing up the genealogies of the circulation in the media space of audiovisual archives; 2. For any shot, digging out the very first video including it, from which all the following videos derive, as it would enable us to compare new contexts (accompanying journalistic discourse, montage with other sequences in a news report, etc.) to the original one, so to understand the decontextualization-recontextualization operated by journalists at each reuse.

Here follows an example of retrieving different occurrences of stock shots starting from key frames representing the Treaty of Rome.

1. New annotation:

During data collection, we have noticed that whenever the event of the creation of the European Economic Community is mentioned, it is probable that the stock shots in black and white showing the signature of a document by a group of politicians at the Palazzo dei Conservatori are used as illustration. Thus, group members who have detected the presence of this specific scene would annotate the stock shots as “Treaty of Rome” as a referred event.

2. Aggregation according to annotation:

Next, we have managed to aggregate all the data marked as “Treaty of Rome” in our initial corpus into a separate group. For French TV news data for example, we find 43 occurrences (different stock shots). Most of the frames seem highly similar and probably come from the same video source.

3. User selection for the initial search:

A file containing all the occurrences was then shared with our project members based in INA in Paris. A manual job is needed in the first place to select a few “good” pictures initialising the automated visual search. As a matter of fact, slightly different screenshots may have been selected from the same stock and therefore from the same video source. This is affected both from journalistic video editing and from our collecting practices (despite common guidelines for annotators).

4. First proposal by the machine:

Automate visual recognition is conducted on our initial corpus plus the new import of Web videos increasing its scope. The machine proposes a series of key frames which are visually like the scene of the signature as it is represented by our input of screenshots.

5. User validation:

Based on the first proposal made by the machine, the user iteratively tweaks this result by selecting and deselecting screenshots that do or do not visually belong to the footage looked for.

Figure 9

Figure 10

6. Modelling of user-supplied knowledge:

The software will then use the new selection made by the user to increase the fit of the initial search model. With the renewal of the model, a second series of frames from different data sources are proposed. Once again, the user must tell the machine whether one screenshot is a real or a false positive. One of the key technological traits of Snoop is relevance looping, which allows for the creation of custom models for the visual entities, continuously amending the working models by including user input.

7. Exhaustive search of corpus:

During our first experimentation with Snoop, we managed to dig out a total of 119 occurrences of the stock shot about the signature of the Treaty of Rome. At the end of the process, Snoop provides us with a file in JSON format to download, in which the metadata of each positive result are contained. In this way, we can trace back the found footage to the video source where each of its occurrences is used and, more importantly, we increased the exhaustivity of our video corpus beyond the initial one (obtained through text queries).

8. By applying the same procedure to all the historical stock shots in our corpus, we aim to trace all the occurrences of these video sequences, grounding a further step of close reading: examining recontextualization of every reuse, looking for trends and patterns. It should be noted that a significant qualitative research work accompanies the whole process of retrieving the reusage/circulation of visual data in our corpus (and will keep accompanying the analytical phase). Here, we admit that a limitation of Snoop is that it retrieves the key frame of a sequence at a two-second interval. There is a possibility that Snoop finds us certain key frames from the same occurrence of a sequence. However, this kind of redundancy only occupies a small part in our case, since most stock shots are very short, usually lasting several seconds, so probably less than two seconds for a specific frame. Therefore, we consider Snoop as a relevant and reliable tool for retrieving good results, although a validation by humans will still be needed in a later step.

The larger our corpus is, the more metamorphoses of stock shots we can possibly find. With the help of a machine, we have also managed to retrieve certain relevant occurrences existing in our initial corpus but were missed by researchers during manual data collection. However, larger quantities may not be able to serve our second objective, that is tracing stock shots all the way back to their very first video source. In metaphorical terms, even if we could capture all the repetitions, we may lack access to the archived original incarnation of the “ghost” that is the video source, simply because it doesn’t exist in the databases accessible to us (including INA, Rai Teche, Mediaset, Web archives, and the live web). In this case, our solution is to rebuild the “ghost” ourselves, starting from the data we have, by using the fragments in our collection that look highly similar and thus very possibly come from the same source.

Figure 11

Anyway, in some cases, the original video source is available online or happens to be stocked in the database of our archival partners, and Snoop may effectively help us to find it. Sometimes, an even simpler textual search is sufficient for retrieving a certain source. For example, for the sequence of the signature of the Treaty of Rome, we have tried to search for the original video in available archival deposits, a copy of which turns out to be stocked on RaiPlay, the multimedia portal owned by RAI. Another example is the shot where a customs barrier is falling and being burnt out. In fact, in our corpus, the events related to the history of the Schengen agreement are often illustrated by occurrences of this stock shot. On the platform of EUscreen, we have managed to trace back the original video, whose initial title was “Strasburgo. manifestazione per l’Unione europea” (demonstration for the European Union in Strasbourg) provided by Istituto Luce. This video was broadcast on the 24/08/1950 by La Settimana Incom, an Italian newsreel, distributed weekly in cinemas from 1946 to 1965. According to the metadata accompanying the video document, the initial context of the sequence is about an event in which “a group of French and German students meets on the border to break ancient barriers and arrives in Strasbourg, where there are other young people coming from other European countries: they all demand the unity of the continent,” which happened long before the signature of the Schengen agreement, as such even longer before the concrete abolishment of national borders in Europe. Therefore, we will compare the initial context with its variations evolving at each reuse of the stock shot, to investigate how the significance attributed to a specific sequence has transformed from a student demonstration to, till now, a symbol of a Europe without borders.

It is very common and well documented that stock footage and photos end up used in contexts which may be extremely different from the original one. We wonder which is the theoretical pertinence of this recontextualization. In fact, from a semiotic and communicational perspective (inquiring into the sense effects produced by video content in the media), the original context of some footage is totally not useful, as meaning is construed by situating the video within a larger journalistic discourse. On the other hand, ethical and legal considerations are instead perfectly valid: should some content be reused without any reference to its original intentionality? A complementary research question could therefore be asked: do images and videos have a “model viewer” comparable to what Eco³⁵ calls “model reader” for texts, that is, some trace of the correct interpretation embedded within the cultural object itself and pointing to its intentionality, or instead images and video sequences have a weaker anchoring to their initial intention of use, and correspond more precisely to words and phrases, which can acquire an indefinite number of new meanings in new contexts? As far as we are observing, it is the news items (including a series of possibly very different shots, voice over, texts, etc.) which allow for the identification of an intention to show and to say, and not at all the individual shots.

Figure 12

7 Conclusion

In this article, we first presented the process of corpus construction of the CROBORA project, and highlighted a theoretical question, crossing semiotics and history, about how to reach some missing source (that we called ghost data) from several signs of it. Then we described three different types of stock shots we found in our collection, the illustrative, the historical and the recent. We then focused on only one type, the historical stock shots, which is the most interesting to us although it applies to a relatively small number of stock occurrences. At last, we quickly presented the software Snoop, with the help of which we proceeded to increase the scope of our corpus, and we also aimed for reaching the original source of our stock footage.

It is fascinating to trace the reuses of certain born digital or digitalized artifacts to draw out a genealogy of their trajectory. As suggested by Franco Moretti, we can define a unit of analysis, such as a concept, a device, a trope, a limited narrative unit, and then follow its metamorphoses in a variety of environments³⁶. In this way, significant patterns that would not otherwise be visible may possibly emerge³⁷. In our case, it is first and foremost the audiovisual sequences that have been sought for in the digital environment, with a combination of qualitative and computational means. Later, we will be looking for patterns of repetition possibly emerging from our data, in the hope to discover some patterns in cultural memory, linking the way in which we watch historical events to the way in which we think of them.