Table 1
Mapping between the KOTE Dataset Emotion Classification and Woo-kyu Kang’s Emotion Classification (adapted from Kang, 2024).
| KOTE | KANG |
|---|---|
| 불평/불만(dissatisfaction), 지긋지긋(fed up), 의심/불신(distrust), 한심함(pathetic), 짜증(irritation), 어이없음(preposterous) | 불만(dissatisfaction) |
| 감동/감탄(admiration), 고마움(gratitude), 존경(respect) | 감동/감사(admiration/gratitude) |
| 환영/호의(welcome), 안심/신뢰(relief) | 신뢰/호의(trust/favour) |
| 슬픔(sadness), 힘듦/지침(exhaustion), 서러움(sorrow) | 슬픔(sadness) |
| 화남/분노(anger), 역겨움/징그러움(disgust), 증오/혐오(contempt), 경악(shock) | 분노/혐오(anger/disgust) |
| 기대감(expectancy), 신기함/관심(interest) | 기대(expectation) |
| 아껴주는(care), 흐뭇함(귀여움/예쁨)(attracted) | 애정(affection) |
| 기쁨(joy), 뿌듯함(pride), 편안/쾌적(comfort), 즐거움/신남(excitement), 행복(happiness) | 기쁨(joy) |
| 우쭐댐/무시함(arrogance) | 우쭐댐/무시함(arrogance) |
| 안타까움/실망(disappointment) | 안타까움/실망(disappointment) |
| 불쌍함/연민(compassion) | 불쌍함/연민(compassion) |
| 부끄러움(shame), 절망(despair), 패배/자기혐오 (defeat/self-hatred), 죄책감(guilt) | 수치/죄책(shame/guilt) |
| 공포/무서움(fear), 불안/걱정(anxiety) | 두려움(fear) |
| 없음(NO EMOTION) | 없음(none) |
| 깨달음(realization) | 깨달음(realization) |
| 당황/난처(embarrassment), 부담/안_내킴(reluctant) | 당황/난처(embarrassment/reluctant) |
| 비장함(resolute) | 비장함(resolute) |
| 놀람(surprise) | 놀람(surprise) |
Table 2
Example of the Pre-existing Sohyeonseongrok Emotion Annotation Data (Yoon’s data).
| ID | TEXT | EMOTION CLASSIFICATION |
|---|---|---|
| 1 | 송나라 태종(太宗) 때 강릉후 겸 구석(九錫) 참지정사(參知政事) 용도각(龍圖閣) 태학사(太學士)인 소공의 이름은 경이고, 자는 자문이었다. During the reign of Emperor Taejong of Song, Lord So, who served as Marquis of Gangneung, Chancellor with Nine Bestowals, and Grand Scholar of the Dragon Pavilion, was named Gyeong, with the courtesy name Jamun. | none |
| 2 | 그는 날 때부터 산천이 지닌 기운과 해와 달의 정기, 그리고 천지의 조화를 타고났다. From birth, he was endowed with the energy of mountains and rivers, the essence of the sun and moon, and the harmony of heaven and earth. | expectation |
| 3 | 가슴에는 세상을 다스릴 뜻을 품었고 얼굴에는 어지러운 나라를 평안하게 할 재주가 어려 있었으며, 빼어난 문장과 뛰어난 절개가 당대에 으뜸이었고 수려한 풍채는 천고에 따를 자가 없을 정도였다. In his heart, he harbored the ambition to govern the world; his face reflected the talent to pacify a chaotic nation; his outstanding writing and unyielding integrity were paramount in his era, and his handsome appearance was unmatched throughout history. | expectation |
Table 3
Example of the Pre-existing Sohyeonseongrok Emotion Prediction Data Generated by Model v.1.1.1.
| ID5 | ADMIRATION/GRATITUDE | EXPECTATION | JOY | REALIZATION | SURPRISE | EMBARRASSMENT/RELUCTANT | FEAR | ANGER/DISGUST | DISSATISFACTION |
|---|---|---|---|---|---|---|---|---|---|
| 1 | .025 | .014 | .013 | .019 | .011 | .015 | .013 | .015 | .014 |
| 2 | .672 | .029 | .028 | .201 | .012 | .011 | .017 | .023 | .018 |
| 3 | .85 | .034 | .02 | .044 | .017 | .016 | .019 | .045 | .026 |
| ID | compassion | resolute | shame/guilt | sadness | trust/favour | disappointment | affection | none | arrogance |
| 1 | .011 | .016 | .012 | .016 | .019 | .015 | .009 | .917 | .011 |
| 2 | .021 | .022 | .034 | .033 | .038 | .029 | .023 | .051 | .027 |
| 3 | .018 | .022 | .038 | .031 | .033 | .021 | .021 | .099 | .024 |
Table 4
Example of Sohyeonseongrok Consensus-Labeled Data.
| ID | TEXT | V.1.1.1 | YOON | KWON | LABEL COMPARISON | CONSENSUS-LABEL | CONSENSUS_METHOD |
|---|---|---|---|---|---|---|---|
| 1 | 송나라 태종(太宗) 때 강릉후 겸 구석(九錫) 참지정사(參知政事) 용도각(龍圖閣) 태학사(太學士)인 소공의 이름은 경이고, 자는 자문이었다. During the reign of Emperor Taejong of Song, Lord So, who served as Marquis of Gangneung, Chancellor with Nine Bestowals, and Grand Scholar of the Dragon Pavilion, was named Gyeong, with the courtesy name Jamun. | none | none | none | C=D=E | none | majority_consensus |
| 2 | 그는 날 때부터 산천이 지닌 기운과 해와 달의 정기, 그리고 천지의 조화를 타고났다. From birth, he was endowed with the energy of mountains and rivers, the essence of the sun and moon, and the harmony of heaven and earth. | admiration/gratitude | expectation | none | C≠D≠E | admiration/gratitude | researcher_resolution |
Table 5
Example of Sossi-Samdaerok Consensus-Labeled Data.
| ID | TEXT | V.1.1.1 | V.1.1.2 | V.1.1.3 | V.1.1.4 | V.1.1.5 | V.1.1.6 | V.1.1.7 | V.1.1.8 | LABEL COMPARISON | CONSENSUS-LABEL | CONSENSUS_METHOD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 대송 시절에 승상 소현성의 이름은 경이고 자(字)는 자문이었다. During the Great Song Dynasty, Chancellor So Hyeon-seong’s given name was Gyeong and his courtesy name was Jamun. | none | none | none | none | none | none | none | none | v.1.1.1 = v.1.1.2 = v.1.1.3 = v.1.1.4 = v.1.1.8 = v.1.1.5 = v.1.1.6 = v.1.1.7 | none | majority_consensus |
Table 6
Emotion Prediction Agreement Rates Among Models.
| MODEL | V.1.1.1 | V.1.1.2 | V.1.1.3 | V.1.1.4 | V.1.1.5 | V.1.1.6 | V.1.1.7 | V.1.1.8 |
|---|---|---|---|---|---|---|---|---|
| v.1.1.1 | 100 | 31.97 | 43.8 | 45.42 | 44.14 | 46.91 | 51.34 | 48.31 |
| v.1.1.2 | 31.97 | 100 | 35.79 | 45.49 | 54.72 | 33.18 | 46.44 | 39.26 |
| v.1.1.3 | 43.8 | 35.79 | 100 | 52.09 | 40.47 | 56.68 | 50.19 | 60.97 |
| v.1.1.4 | 45.42 | 45.49 | 52.09 | 100 | 52.41 | 52.78 | 60.16 | 55.52 |
| v.1.1.5 | 44.14 | 54.72 | 40.47 | 52.41 | 100 | 46.81 | 59.39 | 46.34 |
| v.1.1.6 | 46.91 | 33.18 | 56.68 | 52.78 | 46.81 | 100 | 53.87 | 56.58 |
| v.1.1.7 | 51.34 | 46.44 | 50.19 | 60.16 | 59.39 | 53.87 | 100 | 52.22 |
| v.1.1.8 | 48.31 | 39.26 | 60.97 | 55.52 | 46.34 | 56.58 | 52.22 | 100 |
Table 7
Number and Rate of Agreements between Individual Model Predictions and Provisional Multi-Model Consensus Labels.
| MODEL | TRAINING DATA VOLUME | MATCHES | VALID SENTENCES (MATCHED BY FIVE OR MORE) | AGREEMENT RATE (%) |
|---|---|---|---|---|
| v.1.1.1 | 4,308 | 6,705 | 8,635 | 77.65% |
| v.1.1.2 | 5,282 | 5,683 | 8,635 | 65.81% |
| v.1.1.3 | 5,282 | 7,298 | 8,635 | 84.52% |
| v.1.1.4 | 10,564 | 7,740 | 8,635 | 89.64% |
| v.1.1.5 | 9,590 | 7,019 | 8,635 | 81.29% |
| v.1.1.6 | 9,590 | 7,324 | 8,635 | 84.82% |
| v.1.1.7 | 14,872 | 7,867 | 8,635 | 91.11% |
| v.1.1.8 | 5,282 | 7,709 | 8,635 | 89.28% |
Table 8
Agreement Rate between Sossi-Samdaerok Consensus-Labeled Data and Eight Models’ Emotion Predictions.
| MODEL | TRAINING DATA COMPOSITION | TRAINING DATA VOLUME | MATCHED COUNT | AGREEMENT RATE (%) |
|---|---|---|---|---|
| v.1.1.1 | Kang’s data | 4,308 sentences | 8,327 | 53.98% |
| v.1.1.2 | Kwon’s data | 5,282 sentences | 6,682 | 43.32% |
| v.1.1.3 | Yoon’s data | 5,282 sentences | 8,670 | 56.20% |
| v.1.1.4 | Kwon’s & Yoon’s data | 10,564 sentences | 9,126 | 59.16% |
| v.1.1.5 | Kang’s & Kwon’s data | 9,590 sentences | 8,359 | 54.19% |
| v.1.1.6 | Kang’s & Yoon’s data | 9,590 sentences | 8,962 | 58.10% |
| v.1.1.7 | Kang’s & Kwon’s & Yoon’s data | 14,872 sentences | 9,490 | 61.52% |
| v.1.1.8 | Sohyeonseongrok consensus-labeled data | 5,282 sentences | 9,525 | 61.75% |
Table 9
Examples of Researcher Verification and Modification of Deep Learning Model Consensus Results.
| ID | TEXT | V.1.1.1 | V.1.1.2 | V.1.1.3 | V.1.1.4 | V.1.1.5 | V.1.1.6 | V.1.1.7 | V.1.1.8 | CONSENSUS-LABEL |
|---|---|---|---|---|---|---|---|---|---|---|
| 369 | 방생은 어떤 놈인가? What kind of scoundrel is Mister Bang? | none | expectation | expectation | expectation | expectation | expectation | expectation | expectation | dissatisfaction |
| 854 | “네가 옛날 악래(惡來)보다 더하구나. You are stronger than Akrae (惡來) of old. | anger/disgust | admiration/gratitude | anger/disgust | anger/disgust | anger/disgust | anger/disgust | admiration/gratitude | anger/disgust | admiration/gratitude |
| 1760 | 위씨ㆍ강씨ㆍ형씨 세 부인이 각각 그 지아비의 풍모를 보니 그들이 공주의 눈에 들 것이 뻔하였다. When the three wives, Wi, Kang, and Hyeong, each saw the appearance of their husbands, it was obvious that they would catch the princess’s eye. | dissatisfaction | arrogance | expectation | joy | expectation | expectation | expectation | expectation | fear |