Construction of Consensus-Labeled Data for the Study of Emotions in Korean Full-Length Novels

(1.1) Methodological Landscape and Limitations of Emotion Analysis in Korean Full-Length Novels

Computational emotion analysis can support the investigation of literary genres and narrative structures (Kim et al., 2017). However, applying such approaches to Korean full-length novels requires the construction of emotion data suitable for computational analysis, a process that raises particular methodological challenges because of the scale and narrative complexity of these texts. Korean full-length novels feature extensive, multi-generational narrative structures. While traditional close reading can capture complex psychological shifts and Confucian conflicts, applying this qualitative approach consistently across such extensive texts is difficult in practice. Moreover, individual researchers’ emotion annotations may reflect researcher-specific interpretive perspectives, which can contribute to variation across label sets. As recent NLP studies have noted, emotion annotation in textual data may show low inter-annotator agreement because it relies heavily on subjective human interpretation (Öhman, 2020).

Conversely, quantitative deep learning models trained on modern-language datasets (e.g., the Korean Online That-gul Emotions (KOTE) Dataset (Jeon et al., 2022)¹) may have difficulty capturing the archaic syntax and complex discourse contexts characteristic of Korean full-length novels. Historical literary texts can pose challenges for models trained on modern-language data because of linguistic change, archaic vocabulary, and period-specific rhetoric (İlter et al., 2026). In Korean full-length novels, emotion interpretation may additionally depend on broader narrative and ethical contexts, and such information may not always be fully recoverable from an isolated sentence.

(1.2) Construction of Consensus-Labeled Data to Overcome Data Scarcity

Due to the extensive length of Korean full-length novels and the limited number of researchers engaged in digital humanities research on these texts, only a small number of manually annotated emotion datasets are currently available, including Kang’s data,² Kwon’s data, and Yoon’s data.³ While data augmentation and synthesis can increase the volume of training data (Bayer et al., 2023; Vu et al., 2022), increasing data quantity alone does not necessarily resolve differences in annotation criteria or ensure semantic compatibility with historical literary texts. These limitations are particularly relevant to emotion annotation, where interpretations may vary across annotators and depend on narrative and historical context.

Consensus labeling provides a practical means of integrating multiple annotations in domains where individual judgments may differ. Rather than treating one annotation as inherently authoritative, a consensus procedure identifies shared patterns across multiple sources of annotation evidence. Previous studies have shown that disagreement is a persistent feature of emotion annotation and that consensus-based approaches can be useful for reconciling heterogeneous labels (Öhman, 2020; Shoer & Erzin, 2025). However, conventional consensus-building generally depends on multiple human annotators, which can require substantial time and labor when applied to the full texts of large-scale classical novels. Accordingly, this study proposes a methodology for constructing consensus-labeled data by comparing and reconciling existing human annotations with predictions generated by multiple deep learning models.

This methodology builds on a series of the author’s previous studies. The 18-category emotion classification system, developed by reorganizing the 44 KOTE emotion categories to reflect the characteristics of Korean full-length novels, was established in Kang (2024). Based on this classification system, Kang’s data was constructed, and the deep learning emotion prediction ‘model v.1.1.1’⁴ trained on this dataset was developed and evaluated in Kang and Kim (2024). The sentence-level agreement patterns among the three Sohyeonseongrok label sets used in the present study—predictions from model v.1.1.1, Yoon’s data, and Kwon’s data—as well as the interpretation of these differences in terms of readers’ horizons of expectation, were previously discussed in Kang (2025a). Building on these findings, the present study proceeds from the premise that individual researcher annotations need not be treated as representing a single correct interpretation of emotion and that deep learning model predictions can also serve as one source of annotation evidence.

On this basis, two complementary consensus-labeling procedures were developed. For Sohyeonseongrok, the original annotation columns from the previously released label sets were retained, while a new consensus label was derived through majority-rule aggregation and researcher verification. Thus, the present study extends the previous comparison of heterogeneous human and model annotations into a consensus-based data-construction procedure. For Sossi-Samdaerok, emotion predictions generated by model v.1.1.1 and genre- and character-related emotional patterns were previously analyzed in Kang (2025b). Building on this prior work, the present study extends the earlier single-model analysis by constructing seven additional emotion prediction models, applying all eight models to Sossi-Samdaerok, and deriving consensus labels through a multi-model majority-rule procedure followed by researcher verification. Thus, the primary contribution of the present study lies in constructing and evaluating two complementary consensus-labeling procedures: a three-source consensus combining two human annotations and one AI-model prediction for Sohyeonseongrok, and an eight-model majority consensus for Sossi-Samdaerok, both followed by researcher verification. This approach can contribute to mitigating the influence of individual annotator subjectivity, supplementing machine predictions with literary context that is difficult for models to capture, and supporting a transparent and reproducible framework for future research in classical literature and digital humanities.

(3.1) Adoption of an 18-Category Emotion Classification System for Korean Full-Length Novels

As Öhman (2020) points out, there is no universally accepted taxonomy of human emotions, and different annotation schemes must be proposed depending on specific research objectives and textual modalities. To overcome the categorization errors experienced by previous emotion analysis studies that predominantly adopted modern language-based emotion dictionaries, this study utilized Woo-kyu Kang’s 18-category emotion classification system (Kang, 2024) as the label standard. This system was adopted because it was developed to reflect emotion categories relevant to Korean full-length novels, including Samdaerok-type narratives.

Kang (2024) developed this system by reorganizing KOTE’s 44 emotion categories (Jeon et al., 2024) into 18 categories through quantitative analysis and qualitative review of emotion classifications relevant to Korean full-length novels. The mapping is presented in Table 1.

(3.2) Construction of Consensus-Labeled Data for Sohyeonseongrok

For the first target text, Sohyeonseongrok (5,282 sentences), a three-party consensus method involving one deep learning model and two human annotators was adopted.

(3.3) Construction of Multi-Model Consensus-Labeled Data for Sossi-Samdaerok

For Sossi-Samdaerok, which consists of 15,426 sentences, this study applied a multi-model majority-rule approach to reduce the burden of manual annotation by integrating emotion predictions from multiple models.

(4.1) Influence of Annotator-Specific Perspectives on Model Predictions

The multi-model comparison suggests that deep learning models may reflect annotator-specific perspectives present in their training data. As a starting point for examining this pattern, it is useful to consider the differences previously observed among the three Sohyeonseongrok label sets used in the present study.

According to Kang (2025a), among the three Sohyeonseongrok label sets—predictions from model v.1.1.1, Yoon’s data, and Kwon’s data—all three labels matched in 22.30% of the sentences, whereas all three labels differed in 36.88%. The pairwise agreement rates were 33.96% between model v.1.1.1 and Yoon’s data, 32.64% between model v.1.1.1 and Kwon’s data, and 41.12% between Yoon’s and Kwon’s data. Kang (2025a) interpreted these differences as reflecting distinct horizons of expectation represented in the three label sets.

To examine the relationship among the three label sets beyond simple agreement rates, the present study calculated Fleiss’ Kappa for the three emotion label sources—predictions from model v.1.1.1, Yoon’s data, and Kwon’s data. The resulting inter-source agreement coefficient was 0.2978, indicating limited agreement among the three label sets. As Öhman (2020) notes, agreement in emotion annotation can be constrained by the overlap among emotion categories and variation in annotation judgments. The observed level of agreement provides quantitative evidence of the potential influence of annotator- or model-specific perspectives on emotion labeling.

The eight models trained on Kang’s, Kwon’s, and Yoon’s data, their combinations, or the consensus-labeled data derived from these annotation sources showed variation in their emotion prediction agreement rates. This suggests that annotator-specific perspectives reflected in the training data may influence model predictions.

To quantify the overall level of agreement among the predictions of the eight models, Fleiss’ Kappa was calculated. The resulting coefficient was 0.4390, indicating a moderate level of agreement among the models. When considered together with the variation in pairwise agreement rates shown in Table 6, this result indicates that model predictions varied according to training-data configuration and is also consistent with the possibility that annotator-specific perspectives reflected in the training data may influence emotion predictions.

To examine the relationship between training-data configuration and model agreement more systematically, the mean agreement rate of each model with the other seven models was calculated. Among the three models trained on single annotation datasets, the mean agreement rates were 44.56% for model v.1.1.1, 40.98% for model v.1.1.2, and 48.57% for model v.1.1.3, with an overall mean of 44.70%. By comparison, models v.1.1.4, v.1.1.5, and v.1.1.6, each trained on a combination of two annotation datasets, showed mean agreement rates of 51.98%, 49.18%, and 49.54%, respectively, with an overall mean of 50.23%. Model v.1.1.7, trained on the combination of all three annotation datasets, showed the highest mean agreement rate among the eight models at 53.37%. These comparisons suggest two main implications.

First, models trained on more diverse annotation sources tended to show higher mean agreement with the other models. However, this tendency may be associated not only with greater diversity of annotation sources but also with increased training-data volume. Because the number of annotation sources and the amount of training data increased simultaneously in the present model configurations, the individual effects of these two factors cannot be separated on the basis of the present comparison alone.

Second, model v.1.1.8, trained on the Sohyeonseongrok consensus-labeled data, showed a level of agreement comparable to that of models trained on larger combinations of annotation datasets. This does not mean that consensus-labeled data consistently outperform combined training data; rather, it suggests that training data refined through a consensus process may provide a viable alternative to simply increasing the volume of training data.

Taken together, these results are consistent with the view that emotion labels are not fixed correct answers but may reflect annotator-specific interpretive perspectives. Deep learning models trained on such labels may likewise reproduce aspects of those perspectives. Accordingly, comparing predictions across multiple models can help reveal not only differences in emotion recognition across annotation sources but also areas of shared recognition among them.

(4.2) Value of Consensus-Labeled Data as an Efficient Training Resource

Section 4.1 examined how annotator-specific perspectives reflected in the training data may influence model predictions and how model predictions may vary according to the composition of the training data. In general, models trained on multiple datasets showed higher agreement rates with other models than those trained on a single dataset, with model v.1.1.7, trained on the combined data of Kang, Kwon, and Yoon, showing the highest agreement rate. However, model v.1.1.8, trained on the Sohyeonseongrok consensus-labeled data, also showed an agreement rate with the other models comparable to that of model v.1.1.7. This pattern suggests the potential efficiency of consensus-labeled data as a training resource. To examine this possibility more closely, this section focuses on a comparison between models v.1.1.7 and v.1.1.8 and considers the value of consensus-labeled data as an efficient training resource.

Model v.1.1.7 was trained on 14,872 sentences obtained by combining Kang’s, Kwon’s, and Yoon’s annotation datasets, whereas model v.1.1.8 was trained on 5,282 Sohyeonseongrok consensus-labeled sentences constructed by reconciling differences among the three annotation sources through majority-rule consensus and researcher verification. Table 7 shows the extent to which each model’s emotion predictions agree with the provisional model-consensus labels derived through majority voting across the eight models prior to researcher verification. Accordingly, the agreement rates in Table 7 should be understood as measures of consistency with the multi-model consensus rather than as independent estimates of predictive accuracy. As shown in Table 7, model v.1.1.7 showed the highest agreement with the provisional multi-model consensus labels at 91.11%, while model v.1.1.8 showed a comparable agreement rate of 89.28%. These results do not indicate that consensus-labeled data outperform combined training data. Rather, it is noteworthy that model v.1.1.8, trained on approximately one-third of the training data used for model v.1.1.7, achieved a comparable level of agreement. This pattern suggests that, in addition to the quantitative expansion of training data, the way in which differences among annotation sources are reconciled may also be associated with consistency in model predictions.

A similar pattern was observed when the predictions of each model were compared with the final consensus labels after researcher verification. Table 8 evaluates the agreement between each model’s predictions and the final consensus labels across all 15,426 sentences in the Sossi-Samdaerok dataset.

As shown in Table 8, model v.1.1.7 showed an agreement rate of 61.52% with the final Sossi-Samdaerok consensus labels, whereas model v.1.1.8 showed a slightly higher agreement rate of 61.75%, a difference of only 0.23 percentage points. This pattern differs from the comparison with the provisional model-consensus labels before researcher verification, where model v.1.1.7 showed the higher agreement rate. However, this very small difference should not be interpreted as evidence that model v.1.1.8 is superior. Rather, the more important point is that model v.1.1.8, trained on 5,282 Sohyeonseongrok consensus-labeled sentences, achieved a level of agreement with the final consensus labels that was virtually identical to that of model v.1.1.7, which was trained on 14,872 sentences obtained by combining three annotation datasets. In the context of Korean full-length novel research, where annotated emotion data remain limited and the construction of large-scale new qualitative annotation datasets requires substantial time and resources, this finding suggests that reconciling differences among existing annotation sources through consensus and researcher verification may provide a practical alternative to relying solely on continued quantitative expansion of annotated training data.

(4.3) Context-Dependent Emotion Interpretation and the Role of Researcher Verification

The preceding section showed that consensus-labeled data may serve as an efficient training resource by achieving a comparable level of consistency across model predictions even with a relatively smaller volume of training data. However, consensus across multiple models did not always correspond to interpretations based on the surrounding narrative context. During the construction of the Sossi-Samdaerok consensus-labeled dataset, provisional model consensus was reached for 8,635 sentences in which five or more of the eight models predicted the same emotion. After reviewing the preceding and subsequent narrative context, the researcher revised the consensus labels for 297 of these sentences (approximately 3.44%). This indicates that majority-rule consensus among models can be used to derive emotion labels efficiently, while in some cases the resulting labels may not sufficiently reflect the broader narrative context, making researcher verification a useful complementary step. Because the models received one sentence at a time without adjacent-sentence context, these discrepancies should also be interpreted in relation to the sentence-level input design rather than as an inherent limitation of transformer-based models.

Table 9 presents examples in which provisional model-consensus labels were revised through researcher verification. For ID 369, seven of the eight models predicted ‘expectation.’ However, when the utterance is examined in its surrounding narrative context, it can be interpreted not simply as a question seeking information about Mr. Bang, but as an expression of Sohyeonseong’s hostility and dissatisfaction toward him for having put Lady Wi in danger. Accordingly, the researcher assigned ‘dissatisfaction’ as the final emotion label. This case suggests that surface-level formal features of a sentence, such as an interrogative structure, do not necessarily correspond to the emotion conveyed in its narrative context.

Next, for ID 854, six models identified the emotion as ‘anger/disgust.’ Akrae (惡來)⁶ has traditionally been represented as a negatively evaluated historical figure while also serving as a conventional figure associated with extraordinary physical strength. This historical and cultural ambiguity may have influenced the models’ emotion predictions. However, when the utterance is read together with the immediately following sentence, ID 855, “Thereafter, [he] cherished Un-seong very affectionately,” the comparison with Akrae can be understood not as a criticism of Un-seong but as a positive evaluation of his strength. Accordingly, the researcher assigned ‘admiration/gratitude’ as the final emotion label. This case suggests that the speaker’s evaluative stance may not always be adequately understood from the lexical or surface meaning of an individual sentence alone and that adjacent narrative context can provide an important basis for emotion interpretation.

Finally, for ID 1760, five models identified the emotion as ‘expectation.’ In the immediately following sentence, ID 1761, however, the three wives are described as becoming deeply worried and anxious. When this narrative sequence is taken into account, ID 1760 can be interpreted not simply as expressing a positive expectation about the future, but as describing the wives’ recognition of a possible threat to their marital relationships arising from the princess’s interest in their husbands. Accordingly, the researcher assigned ‘fear’ as the final emotion label. This case demonstrates that the emotion conveyed by a particular sentence may become clearer when interpreted in relation to subsequent narrative developments rather than solely on the basis of expressions contained within the sentence itself.

Although these three cases are not intended to represent all 297 instances revised by the researcher, they show that information extending beyond an individual sentence—such as sentence form, the speaker’s evaluative stance revealed in adjacent sentences, and emotions clarified through subsequent narrative development—can influence emotion labeling in literary texts. Emotion interpretation in literary texts may therefore depend not only on expressions within an individual sentence but also on narrative progression, implicit evaluation, historical and cultural meanings, and relations with the surrounding context. The existence of 297 revised cases indicates that researcher verification functioned as a substantive complementary stage to multi-model consensus, while the examples above illustrate more specifically how some of these revisions involved context-dependent emotion interpretation.

In this respect, the consensus-labeling methodology proposed in this study can be understood as a complementary integration of AI-based large-scale text processing and the contextual review of humanities researchers. It offers a practical procedure for considering both computational consistency and contextual interpretation in the construction of literary emotion data.