Evaluating the Evaluator: Problems With SemEval-2020 Task 1 for Lexical Semantic Change Detection
References
- Alatrash, R., Schlechtweg, D., Kuhn, J., & Schulte im Walde, S. (2020). CCOHA: Clean Corpus of Historical American English. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk & S. Piperidis (Eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 6958–6966).
European Language Resources Association .https://aclanthology.org/2020.lrec-1.859/ - Basile, P., Caputo, A., Caselli, T., Cassotti, P., & Varvara, R. (2020).
DIACR-Ita @ EVALITA2020: Overview of the EVALITA2020 Diachronic Lexical Semantics (DIACR-Ita) Task . In V. Basile, D. Croce, M. Maro & L. C. Passaro (Eds.), EVALITA Evaluation of NLP and Speech Tools for Italian – December 17th, 2020 (pp. 411–419). Accademia University Press.https://books.openedition.org/aaccademia/7613 - Cassotti, P., De Pascale, S., & Tahmasebi, N. (2024). Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Types. In L.-W. Ku, A. Martins & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4539–4553).
Association for Computational Linguistics .https://aclanthology.org/2024.acl-long.249/ - Cassotti, P., Siciliani, L., DeGemmis, M., Semeraro, G., & Basile, P. (2023). XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. In A. Rogers, J. Boyd-Graber & N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 1577–1585).
Association for Computational Linguistics .https://aclanthology.org/2023.acl-short.135/ - Chen, J., Chersoni, E., Schlechtweg, D., Prokic, J., & Huang, C.-R. (2023). ChiWUG: A Graph-based Evaluation Dataset for Chinese Lexical Semantic Change Detection. In N. Tahmasebi, S. Montariol, H. Dubossarsky, A. Kutuzov, S. Hengchen, D. Alfter, F. Periti & P. Cassotti (Eds.), Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change (pp. 93–99).
Association for Computational Linguistics .https://aclanthology.org/2023.lchange-1.10/ - Cruse, D. A. (1986). Lexical Semantics. Cambridge University Press.
- Davies, M. (2012). Expanding horizons in historical linguistics with the 400-million word Corpus of Historical American English. Corpora, 7(2), 121–157. 10.3366/cor.2012.0024
- Deutsches Textarchiv (2017). Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache.
http://www.deutschestextarchiv.de/ - Fedorova, M., Mickus, T., Partanen, N., Siewert, J., Spaziani, E., & Kutuzov, A. (2024). AXOLOTL’24 Shared Task on Multilingual Explainable Semantic Change Modeling. In N. Tahmasebi, S. Montariol, A. Kutuzov, D. Alfter, F. Periti, P. Cassotti & N. Huebscher (Eds.), Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change (pp. 72–91).
Association for Computational Linguistics .https://aclanthology.org/2024.lchange-1.8/ - Frermann, L., & Lapata, M. (2016). A Bayesian Model of Diachronic Meaning Change. Transactions of the Association for Computational Linguistics, 4, 31–45.
https://aclanthology.org/Q16-1003/ - Geeraerts, D. (1993). Vagueness’s puzzles, polysemy’s vagaries. Cognitive Linguistics, 4(3), 223–272. 10.1515/cogl.1993.4.3.223
- Geeraerts, D. (2010). Theories of lexical semantics. Oxford University Press. 10.1093/acprof:oso/9780198700302.001.0001
- Geeraerts, D. (2020).
Semantic Change . In The Wiley Blackwell Companion to Semantics (pp. 1–24). John Wiley & Sons, Ltd.https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118788516.sem042 - Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30). 10.1073/pnas.2305016120
- Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016). Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In K. Erk & N. A. Smith (Eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1489–1501).
Association for Computational Linguistics .https://aclanthology.org/P16-1141/ - Hanks, P. (2013). Lexical Analysis: Norms and Exploitations. The MIT Press. 10.7551/mitpress/9780262018579.001.0001
- Kilgarriff, A. (1997). “I don’t believe in word senses”.
https://arxiv.org/abs/cmp-lg/9712006 . _eprint: cmp-lg/9712006. - Kutuzov, A., & Pivovarova, L. (2021). RuShiftEval: A Shared Task on Semantic Shift Detection for Russian. In Computational linguistics and intellectual technologies.
- Kutuzov, A., Pivovarova, L., & Giulianelli, M. (2021). Grammatical Profiling for Semantic Change Detection. In Proceedings of the 25th Conference on Computational Natural Language Learning, (pp. 423–434). Online:
Association for Computational Linguistics .https://aclanthology.org/2021.conll-1.33 - Kutuzov, A., Touileb, S., Mæhlum, P., Enstad, T., & Wittemann, A. (2022). NorDiaChange: Diachronic Semantic Change Dataset for Norwegian. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk & S. Piperidis (Eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 2563–2572).
European Language Resources Association .https://aclanthology.org/2022.lrec-1.274/ - Kutuzov, A., Velldal, E., & Øvrelid, L. (2022). Contextualized embeddings for semantic change detection: Lessons learned. Northern European Journal of Language Technology, 8.
https://aclanthology.org/2022.nejlt-1.9/ - Laicher, S., Kurtyigit, S., Schlechtweg, D., Kuhn, J., & Schulte im Walde, S. (2021). Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In I.-T. Sorodoc, M. Sushil, E. Takmaz & E. Agirre (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (pp. 192–202). Online:
Association for Computational Linguistics .https://aclanthology.org/2021.eacl-srw.25/ - Loureiro, D., D’Souza, A., Muhajab, A. N., White, I. A., Wong, G., Anke, L. E., Neves, L., Barbieri, F., & Camacho-Collados, J. (2022). TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media.
https://arxiv.org/abs/2209.07216 . _eprint: 2209.07216. - Ma, X., Strube, M., & Zhao, W. (2024). Graph-based Clustering for Detecting Semantic Change Across Time and Languages. In Y. Graham & M. Purver (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1542–1561).
Association for Computational Linguistics .https://aclanthology.org/2024.eacl-long.93/ - McGillivray, B., & Kilgarriff, A. (2013).
Tools for historical corpus research, and a corpus of Latin . In P. Bennett, M. Durrell, S. Scheible & R. J. Whitt (Eds.), New Methods in Historical Corpus Linguistics vol. 3 of Korpuslinguistik und interdisziplinäre Perspektiven auf Sprache (pp. 247–257). Narr. - McGillivray, B., Schlechtweg, D., Dubossarsky, H., Tahmasebi, N., & Hengchen, S. (2020). LatinISE test data for SemEval 2020 task 1 with additional token versions of the corpora.
https://zenodo.org/record/3992738 - Oda, K., Takamura, H., Shirai, K., & Kertkeidkachorn, N. (2025). Improving Interpretability of Lexical Semantic Change with Neurobiological Features. In C.-R. Huang, Y. Harada, J.-B. Kim, N. T. Huyen, L. T. Huong, P. Hien, E. Chersoni, L. M. Nguyen, R. E. O. Roxas & S. Dita (Eds.), Proceedings of the 39th Pacific Asia Conference on Language, Information and Computation (pp. 1–14).
Association for Computational Linguistics .https://aclanthology.org/2025.paclic-1.1/ - Phan-Tat, B., Heylen, K., Geeraerts, D., Pascale, S. D., & Speelman, D. (2026a). ReFRAME or Remain: Unsupervised Lexical Semantic Change Detection with Frame Semantics.
https://arxiv.org/abs/2602.04514 . _eprint: 2602.04514. - Phan-Tat, B., Heylen, K., Geeraerts, D., Pascale, S. D., & Speelman, D. (2026b). Transparent Semantic Change Detection with Dependency-Based Profiles.
https://arxiv.org/abs/2601.02891 . _eprint: 2601.02891. - Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., & Tahmasebi, N. (2020). SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation (pp. 1–23).
International Committee for Computational Linguistics .https://aclanthology.org/2020.semeval-1.1 - Schlechtweg, D., Schulte im Walde, S., & Eckmann, S. (2018). Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change. In M. Walker, H. Ji & A. Stent (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) (pp. 169–174).
Association for Computational Linguistics .https://aclanthology.org/N18-2027/ - Språkbanken Text. (2019). The Kubhist Corpus, v2.
https://spraakbanken.gu.se/en/resources/kubhist2 . Published: Data set. - Språkbanken Text. (2024). SemEval2020 Task 1.
https://spraakbanken.gu.se/en/resources/semeval2020 - Staatsbibliothek zu Berlin. (2018a). Berliner Zeitung Diachronic Newspaper Corpus.
- Staatsbibliothek zu Berlin. (2018b). Neues Deutschland Diachronic Newspaper Corpus.
- Tahmasebi, N., Borin, L., Jatowt, A., Xu, Y., & Hengchen, S. (Eds.) (2021). Computational approaches to semantic change. No. 6 in Language Variation. Language Science Press.
- Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., & Liu, H. (2024). Large Language Models for Data Annotation and Synthesis: A Survey. In Y. Al-Onaizan, M. Bansal & Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 930–957).
Association for Computational Linguistics .https://aclanthology.org/2024.emnlp-main.54/ - Traugott, E. C. (2017). Semantic Change.
https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655.001.0001/acrefore-9780199384655-e-323 - Umarova, K., Lee, L., & Kim, L. (2025). Current Semantic-change Quantification Methods Struggle with Discovery in the Wild. In C. Christodoulopoulos, T. Chakraborty, C. Rose & V. Peng (Eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 35354–35367).
Association for Computational Linguistics .https://aclanthology.org/2025.emnlp-main.1791/ . 10.18653/v1/2025.emnlp-main.1791 - Zamora-Reina, F. D., Bravo-Marquez, F., & Schlechtweg, D. (2022). LSCDiscovery: A shared task on semantic change discovery and detection in Spanish. In N. Tahmasebi, S. Montariol, A. Kutuzov, S. Hengchen, H. Dubossarsky & L. Borin (Eds.), Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change (pp. 149–164).
Association for Computational Linguistics .https://aclanthology.org/2022.lchange-1.16/ - Zhou, W., Tahmasebi, N., & Dubossarsky, H. (2023). The Finer They Get: Combining Fine-Tuned Models For Better Semantic Change Detection. In T. Alumäe & M. Fishel (Eds.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa) (pp. 518–528).
University of Tartu Library .https://aclanthology.org/2023.nodalida-1.52/ - Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024). Can Large Language Models Transform Computational Social Science? Computational Linguistics, 50(1), 237–291.
https://aclanthology.org/2024.cl-1.8/
DOI: https://doi.org/10.5334/johd.547 | Journal eISSN: 2059-481X
Language: English
Page range: 103 - 103
Submitted on: Apr 9, 2026
Accepted on: May 26, 2026
Published on: Jul 31, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
© 2026 Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.