A Classification Benchmark Based on the Literary Theme Ontology

1 Introduction

In this research, we use human-annotated themes in TV series episodes to automatically classify the themes in TV episode summaries and subtitles. The goal is to introduce a new benchmark dataset of summaries and subtitles of TV series episodes, which can be used for theme classification. Additionally, we present baseline results of the theme classification task, using traditional bag-of-words classification methods, such as logistic regression and support vector machine (SVM), as well as different small open-weight large language models (LLM), as they are promising for classification in cultural data when annotations for fine-tuning are not available (Bamman et al., 2024).

The thematic annotations used in this work are drawn from the Literary Theme Ontology (LTO) v2025.04 release, which comprises 2,977 hierarchically organized literary themes and 4,218 thematically annotated stories (Onsjö & Sheridan, 2025a; Sheridan et al., 2019). This taxonomy is used to identify themes in different stories in the form of works of, for example, literature, films, and TV episodes. Using LTO thematic annotations, a new benchmark dataset consisting of themes occurring in 644 TV episode summaries and 956 TV series episode subtitles was created.

Themes can be seen as a higher order label of categories of elements of stories, which can relate stories to each other (Brinker, 1993; Rimmon-Kenan, 1995). As themes are an abstract categorization, we have chosen to work with literary themes lowest in the hierarchy of the LTO. The more general parent themes can easily be retrieved via the Python package totolo (Onsjö & Sheridan, 2025b). Thus this benchmark dataset of TV series episodes can be used not only for research on the specific themes used in this classification task, but also in research on the parent themes.

Our goal is to offer a resource for the automatic classification of themes. In the theme classification task, we use episode summaries and subtitles to predict the ten most frequent themes in the TV episodes. Since multiple themes can occur per episode, this is a multi-label classification task, which is hard to perform due to the limited data available for each theme. Additionally, themes in TV series are often not explicitly expressed in the summaries or subtitles. For example, a theme can also be derived based on prior knowledge about the specific characters discussed: when two characters are married, any episode discussing their relationship can be linked to the theme husband and wife. However, it does not necessarily mean that it is mentioned in the subtitles of the summaries that the couple is married, as it can be assumed that someone watching an episode or reading the summary has seen previous episodes of the TV series, and is already aware of the married status of the characters. These implicit themes increase the complexity of the multi-label classification task. This new dataset of TV series episode summaries and subtitles has a high potential for humanities research on literary themes as it can be extended using the further available LTO annotations as a starting point.

2 Related work

The classification of fiction by theme and subject has been a persistent challenge in library and information science (LIS), knowledge organization, literary studies, and folklore studies. As this benchmark focuses on the textual representation of themes, this section focuses on text-based work.

Early 20th-century LIS scholars, such as E.A. Baker and William Borden, recognized the need for thematic classification to improve reader access, but traditional systems like Dewey Decimal and Library of Congress have historically focused on form, genre, and language rather than thematic content (Baker & Shepherd, 1987). This disciplinary emphasis on formal attributes has limited the ability of readers and researchers to discover fiction based on thematic resonance. Annelise Pejtersen’s work has sought to address these limitations by identifying subject matter, setting, author’s intention, and accessibility as critical dimensions for indexing fiction, advocating for systems that accommodate both denotative and connotative elements (Mark Pejtersen & Austin, 1983). Building on this, Jarmo Saarti introduced the Kaunokki thesaurus, which integrates facets such as events, actors, and spaces to better capture the complexity of fictional themes (Saarti, 2019). These contributions reflect a broader disciplinary shift toward user-centered and multidimensional approaches to classification. A notable advancement comes from the Integrative Levels Classification, which organizes knowledge around phenomena, enabling fiction to be indexed by its thematic content—such as “censorship” or “bravery” (Almeida & Gnoli, 2021). This phenomenon-based approach facilitates hybrid retrieval systems that bridge fiction and nonfiction.

The study of themes in literary studies has evolved through a dynamic interplay of interpretive, theoretical, and interdisciplinary approaches. Traditionally, thematic criticism focused on identifying and analyzing the central ideas or “aboutness” of literary works, often emphasizing the signified—the conceptual or ideological content—over formal or linguistic structures. Despite some skepticism, particularly during the rise of formalist and poststructuralist methodologies, thematic analysis has persisted and even flourished within politically engaged and interdisciplinary frameworks, such as feminist, ethnic, and cultural studies, where themes often serve as lenses for examining broader social and historical contexts (Sollors, 1993). The resurgence of thematic criticism in the late 20th century is evident in collections like Bremond et al. (1995), which seeks to theorize thematics by distinguishing it from related concepts like motif, subject matter, and concept, while also exploring its connections to narratology and structuralist analysis. Similarly, Louwerse and Van Peer (2008) expand the scope of thematic inquiry by integrating cognitive, computational, and cross-disciplinary perspectives, demonstrating how themes function not only in literature but also in discourse, media, and cultural artifacts.

In folklore studies, motif classification has long been a foundational method for analyzing narrative structures and thematic elements (Propp, 1968). The Aarne-Thompson-Uther Index categorizes folktales based on recurring motifs, such as “the hero’s journey” or “the wicked stepmother”, providing a systematic framework for comparative analysis across cultures and historical periods (Uther, 2004). Motifs are distinct, recurring narrative elements in folklore, such as “troll under the bridge” (Yarlott & Finlayson, 2016). Motifs and themes are thus both narrative elements that occur throughout a tale or TV series above the word level (Hagedorn & Darányi, 2022). As motifs in myths are recurring narrative elements throughout texts over time, motifs can be automatically detected using word similarity measures across texts with similar motifs. A zero-shot approach to motif classification consists in classifying motifs based on the highest cosine similarity between sets of texts and corresponding motifs (Matveeva & Malykh, 2022). Another successful approach to cluster motifs is labeled Latent Dirichlet Allocation (LDA) (Karsdorp & van den Bosch, 2013). Lastly, folktales with the same story type, defined as stories with similar recurring plot, motifs and themes, can be successfully classified using a ranking task (Nguyen et al., 2013). This line of research has only worked with written text; an open question is how well the automatic classification of motifs and themes can be performed on audiovisual material like films and TV series. In particular, it is underexplored whether relying only on textual data (e.g., subtitles, scripts, and summaries) related to multimodal artifacts can lead to good results.

The LTO focuses mostly on the thematic classification of films and TV series episodes. Onsjö and Sheridan (2020) show how theme enrichment analysis can be applied to the comparative study of audiovisual content, using the Star Trek franchise as a case study. They statistically identify themes that are significantly over-represented in specific subsets of episodes relative to a broader background set, revealing distinct thematic patterns across different Star Trek series. The analysis not only highlights what makes each series unique but also illustrates how thematic trends evolve over time, offering a quantitative, data-driven approach to comparing narrative content in visual media. Despite the usefulness of thematic annotation for the comparative analysis of fiction, the LTO is one of the few structured thematic resources available and its annotations only cover a limited set of works. It would be extremely useful to implement a system for the automatic identification of these themes. To this end, we present an annotated dataset that can be used to train computational models.

3 The Literary Theme Ontology and thematic annotation

The LTO comprises a hierarchically arranged collection of literary themes together with a trove of thematically annotated works of fiction. In the LTO, a literary theme is broadly conceived as either a topic addressed by a work of fiction or an opinion expressed by the work about that topic (Sheridan et al., 2019). Themes may often be adequately captured at an appropriate level of generality by a single word or short phrase, as in such stock examples as courage, coming of age, and coping with the passage of time. These examples are value-neutral abstractions. Themes may equally take the form of value judgments, such as ignorance is bliss or be wary of strangers. Themes may also represent more complex ideas, for which the short name serves only as a guide and must be supplemented by a fuller definition to be properly understood (e.g. capitalism). The LTO theme inventory and hierarchical structure draw inspiration from prior research across multiple domains, including reference works that curate lists of themes (Armstrong & Armstrong, 2001; McClinton-Temple, 2010; Seigneuret, 1988), online resources that tend to group themes into categories (Encyclopedia of Science Fiction contributors, 2025; TV Trope contributors, 2025; Wikipedia contributors, 2025), and formal theme/motif taxonomies (Bartalesi & Meghini, 2017; Khan et al., 2016).

LTO themes are arranged in a hierarchy, and themes therein can be closely related to each other. The upper levels of the LTO hierarchy are loosely based on a traditional classification system proposed by literary critic William Henry Hudson (Hudson, 1913; Sheridan et al., 2019). At the highest level, the three domain themes the human world, the natural world, and alternate reality descend from the generic root theme literary thematic entity. Each of these domain themes branches into numerous sub-themes. For example, the human world subdivides into personal human experience (i.e., themes concerning individuals, their inner lives, their interactions with others, and their engagement with the surrounding world), human idea about life (i.e., mainly common aphorisms about how to lead a good life), human nature (i.e., claims about inherent aspects of humanity), and society (i.e., themes addressing issues arising from people living together in societies).

According to the LTO guidelines (Onsjö & Sheridan, 2025c), a theme is considered “central” to a story when it is either treated more-or-less continuously throughout the main narrative or plays an important role in the story’s resolution. A theme is deemed “peripheral” when it is featured only briefly. In LTO story annotations, peripheral themes are labeled minor, while central themes are designated either major or choice. Choice themes reflect what the thematic annotator regards as truly significant within the story, and are therefore inherently more subjective than the central–peripheral distinction itself. In brief, the annotators watched the TV series episodes, recorded themes with motivations, and came to consensus by comparing their findings (see Appendix A.1 for an expanded discussion). For a comprehensive treatment of the annotation process see the LTO wiki (Onsjö & Sheridan, 2025c).

To illustrate the differences between the choice, major and minor themes, we will briefly discuss the LTO annotation of the classic fairy tale Little Red Riding Hood (Onsjö & Sheridan, 2025c) which are part of the LTO wiki.¹ The choice themes are beware of strangers, as this theme captures what is generally considered to be the moral of the story, and appearances can be deceiving, as this theme could also argued to be a moral of the story. The major themes are human childhood and wicked character vs. virtuous character, as these themes occur in the majority of the fairy tale. The minor themes are coping with a loved one being gravely ill, grandmother and granddaughter and mother and daughter. Each theme is motivated; for example, the motivation for human childhood is “Little Red Riding Hood innocently trusted the Wolf, and engaged in childish diversions such as chasing butterflies”, and the motivation for coping with a loved one being gravely ill is “The mother was concerned because the grandmother had recently been “very ill””.

4 Dataset description

We derived two different datasets starting from the list of annotated LTO stories, one consisting of summaries collected from Fandom (Fandom, 2025), and one of subtitles. The subtitles were obtained via TVSubs.net (2025) and via VLsub 0.11.1, an extension of the media player VLC that searches for the requested subtitles on Open Subtitles (2025). Due to varying availability of summaries and subtitles, the datasets do not contain the same annotated TV series episodes (see Tables 1 and 2). At the moment of publication, the LTO contained 2070 annotated TV episodes, out of which 1906 with available subtitles and 1345 available summaries. Therefore, the two datasets also contain different themes.

The summaries and subtitles were collected between April 2024 and June 2025. LTO themes have been continuously annotated since 2017 and are ongoing. The benchmark was created using LTO version v2025.04, published on April 14, 2025.

The LTO annotations were the basis for the datasets, with summaries and subtitles collected for the most common themes among the annotated TV episode. We extracted the major and choice themes in TV series episodes from the LTO dataset, and matched them with the available summaries and subtitles. We chose not to consider the minor themes, as they are not consistently present in an episode and therefore are unlikely to be included in summaries, or mentioned in the subtitles. In order to train a theme classifier, a sufficient number of examples is required for each theme. We therefore included only the ten most frequent major themes in the dataset. In this section, we introduce the two datasets and which themes occur in the datasets.

Repository location https://doi.org/10.5281/zenodo.17611141

Repository name Zenodo

Object name The LTO classification benchmark

Format names and versions CSV

Creation dates 2024-08-04 – 2025-06-19

Dataset creators Noa Visser Solissa (University of Groningen) was responsible for conceptualization, data curation, formal analysis, investigation, methodology, Paul Sheridan (University of Price Edward Island) was responsible for conceptualization, data curation and validation and Mikael Onsjö (Independent Researcher) was responsible for data curation.

Language English

License Theme annotations and TV episode summaries: Creative Commons Attribution-Share Alike License 3.0 (Unported) (CC BY-SA); TV episode subtitles: CC-BY

Publication date 2025-11-14

5 Method

To estimate the difficulty of theme classification using these two datasets, we have tested four different classification methods. We have focused on simple, but established, pipelines and local, open-weight LLMs, for replicability and reproducibility purposes. Both datasets are split into a train (80%) and test (20%) set. The train and test sets were randomly created, but with a comparable theme distribution as seen in the full subtitle and summary dataset. The bag-of-words, FastText, and SetFit classification were all trained using the complete train set, whereas the LLM classification uses a zero-shot and a few-shot approach. The code for each classification method can be found on https://github.com/theme-ontology/lto-classification-benchmark.

6 Results

As can be seen in Table 4, the best performing classification models on both datasets is bag-of-words SVM bigrams, for the summaries with 10,000 features and for the subtitles with 5000 features. As can be seen in Tables 5 and 6, the theme time travel has the highest F₁ score for the summary dataset, and husband and wife the highest score for the subtitle dataset. In both cases, these themes also have the highest precision. In both datasets, the theme with the highest recall has a low precision, resulting in a low F₁ score.

Overall, for the bag-of-words logistic regression, SVM, and FastText classification, a pattern can be seen of a high average recall in both datasets, but have a low average precision, meaning that there are many false positives among their classifications. In the summary dataset, the SetFit models have a high precision and a low recall, meaning that the models are accurate in predicting the different classes, but miss many true positives. For the subtitle dataset, the SetFit models have both low precision and low recall, indicating that the models are not successful at predicting the ten themes.

For the LLMs, on the summary dataset the gemma3:12b model approaches the results of the bag-of-words models. For the zero-shot models, all three models predominantly give output in the correct format as shown in the prompt (see Table 9). Thus, the zero-shot models for the summaries are predominantly successful at following the instructions in the prompt. However, the few-shot models are less successful at predicting themes in the requested output, despite having two example input texts and two example outputs. Thus, including examples in the prompt does not result in more predictions using the correct output.

This same pattern is seen for the subtitle dataset. In general, it seems to be much more difficult for the LLMs to follow the instructions when identifying themes in subtitles. Due to these wrongly formatted outputs, the zero-shot and few-shot LLM results have the lowest scores out of all the models used.

In Tables 7 and 8 the best LLM results can be seen. Time travel has the highest F₁ score for the summaries, and murder for the subtitles. The pattern of high precision resulting in a high F₁ score cannot be seen in the best LLM results. The best LLM results do show that some themes are particularly hard to detect, resulting in very low F₁ scores, such as human vs. captivity for the summaries and the desire for vengeance in both datasets.

Table 9 shows the percentages of predictions made by the LLMs that are not a list of strings. When splitting the test set into two subsets of examples below and above the average length of the respective subtitles or summaries, it is seen that almost all predictions with the wrong output are made when the given input text is above the average length of the respective summaries and subtitles. 55.2% of the subtitles test set and 62% of the summaries test set are below average length. Thus, the LLMs are successful at following the instructions in the prompt when an input text is below average.

6.1 Case study

To gain a deeper understanding of the difficulty of the task and the models’ predicted themes, we compare the results for two sample input texts from the test set. Since the theme classifications on summaries obtain the best results, it is most interesting to analyze the summaries and predicted classifications more closely by comparing the SVM classifier predictions to the predictions of the different LLMs. Figures 1 and 2 show two shorter summaries from the summary test set. For the first summary (see Figure 1), the human annotated theme is humanoid robot. As the word “robot” is mentioned five times in the summary and the different parts of the robot, such as the head, torso, and arms, are discussed in the summary, the theme is rather explicitly mentioned in the summary. Humanoid robot has above average F₁ scores across different models, implying that this theme is more easily predictable (see Tables 5 and 7).

Figure 1

Figure 2

The best performing SVM classifier predicts the themes humanoid robot as well as friendship. The zero-shot LLMs gemma3:12b and llama3.1:8b make the same two predictions, whereas mistral:7B predicts humanoid robot and greed for riches. For the few-shot LLMs, gemma3:12b and llama3.1:8b correctly predict humanoid robot without any other themes, whereas mistral:7B again predicts humanoid robot and greed for riches. The prediction greed for riches could be explained by the occurrence of the words “money” and “money-making tool”. Furthermore, the theme greed for riches is actually annotated as a minor theme in the episode in the LTO. Thus, the theme is present in the episode, but not as a major theme, which we are focusing on. The wrongly predicted theme friendship cannot be as explicitly connected to word usage in the texts, but it can be assumed that there are words used that occur in summaries in the train set that do have the theme friendship, as the SVM classifier and the zero-shot gemma3:12b and llama3.1:8b models predict this theme.

In Figure 2, the human-annotated theme is infatuation. This theme has below average results across different models (see Tables 5, 6, 7, 8). The following motivation for the theme is given in the LTO: “Leela and Adlai Atkins” (Onsjö & Sheridan, 2025a). Thus, the relationship between the two characters implies infatuation. In the summary, it is mentioned that Leela had a crush on Adlai Atkins in the past, and that they get into a relationship later in the episode. It is not mentioned that they are infatuated with each other, and based on the summary, romantic love could also be a suitable theme. The SVM classifier predicts the theme friendship and humanoid robot, of which the former misclassification can be explained by the usage of the word “friends” in the summary. In the LTO, the theme humanoid robot does occur in the episode, but as a minor theme (Onsjö & Sheridan, 2025a).

For this summary, the LLMs predict multiple themes. The zero-shot gemma3:12b correctly predicts infatuation, but also predicts friendship and greed for riches. Llama3.1:8b predicts infatuation as well as friendship, husband and wife, humanoid robot and father and son. The zero-shot mistral:7B fails to predict infatuation, but instead predicts the following five themes: romantic love, father and son, greed for riches and human vs. captivity. For the few-shot predictions, gemma3:12b predicts infatuation, romantic love and friendship, llama3.1:8b predicts friendship and father and son and mistral:7B predicts friendship and greed for riches. Father and son is also listed as a minor theme in this episode in the LTO.

As romantic love is the parent of infatuation in the hierarchy of the LTO, it would be logical if the classification models had picked the theme romantic love over infatuation, in particular since the summary explicitly mentions Leela and Adlai Atkins being in a relationship. However, the models seem to be able to distinguish between these two themes in this example, as two models predict infatuation without predicting romantic love and only one model predicts romantic love, without also predicting infatuation.

Thus, this analysis indicates that the predicted themes can be related to specific word usage in the summaries. This also leads to wrongly predicted minor themes, as these themes are explicitly mentioned in the summaries. However, implicit major themes such as infatuation can still be correctly identified by the classifiers. Lastly, despite the theme infatuation being a child of the theme romantic love in the hierarchy of the LTO, the models seem to be able to distinguish between these themes.

7 Discussion

We find that bag-of-words classification is most successful in classifying themes in the summaries and subtitles of TV series episodes. The best results are obtained for the summary dataset, despite the train set being smaller than for the subtitle dataset.

The results for the LLMs also show that the prompt length has a strong influence on the predictions made. All models perform better in the zero-shot prompt, than in the few-shots prompt when example input and output is given. Additionally, the LLMs are more accurate in using the requested output format when the given input text is below average length. Except for one model, all models make no mistakes in answering the prompt in the correct output form, when the given input text is below average for the respective datasets. These results indicate that the LLMs are more likely to hallucinate answers and output formats when the prompt and input texts are longer. Since including examples results in longer prompts, this also explains why the LLMs perform better for the zero-shot prompts than for the few-shot prompts.

Additional File

The additional file for this article can be found as follows:

Competing Interests

The authors have no competing interests to declare.

Author Contributions

Noa Visser Solissa: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Software, Writing – original draft, Paul Sheridan: Conceptualization, Data curation, Validation, Writing – review & editing, Mikael Onsjö: Data curation, Writing – review & editing, Andreas van Cranenburgh: Conceptualization, Supervision, Writing – review & editing, Federico Pianzola: Conceptualization, Supervision, Writing – review & editing.