(1) Overview
Repository location
The datasets are available on Zenodo:
Randomised sentences dataset: https://doi.org/10.5281/zenodo.21223184
Word embeddings generated using fasttext: https://doi.org/10.5281/zenodo.21221575
Word embeddings generated using word2vec: https://doi.org/10.5281/zenodo.21220107
Datasets and additional information are also available via the project website. Both word-embedding datasets can be browsed and downloaded from the project website: https://www.corpas.ie/ga/breis/leabuithe-focal/ or https://www.corpas.ie/en/extras/word-embeddings/.
Context
The three newly presented datasets are by-products that were generated using some or all of the c.150 million words that were collected as part of the Corpas Náisiúnta na Gaeilge1 (Henceforth: CNG) project, Phase 1 of which ran from 2022–2024 inclusive and is the period in question. By-products were generated at the end of this phase, after collection and data cleaning had been completed. This section presents the context of their creation and considerations that have impacted the sharing of these datasets.
The compilation work conducted in the CNG project develops on learnings of the pilot project detailed in Ó Meachair, et al. (2021), which includes considerations for restricted or absent domains and generally limited availability of data in the minority-language context.
The focus of this project was on collecting high-quality data from fluent speakers2 for linguistic analysis, in a way that was similar to Brezina et al. (2021) and Knight et al. (2021a). Source types included in the CNG project were similar to those of the aforementioned corpus-linguistic projects, including: webpages, documents, books, reports and data retrieved from collections or databases using APIs. A limited number of manually transcribed audio and video files were also included. Personal information was anonymised from personal blogs, social media, and any sources that were not published on the Internet but had been submitted for inclusion in the project. Where contact details or addresses are found these have been published on the Internet by organisations, public bodies (governmental, courts, social services, etc), information services (county councils, citizens info, etc), professional services, networking groups (who wish to invite contact), or public outreach activities. Collection was restricted to sources that were known to contain high-quality examples of language use in each genre, an important consideration in the context of a small language community of which a significant percentage are learners but where the scale of the data prevents manual checking of every document. With this approach it is accepted that fewer words would be collected overall and that the resource would be better for it. Verifying the quality of every sample or document manually would have been overly time-consuming and impractical within the three years of the project, and prone to bias-based inaccuracies as well. The CNG compilation work was done with a view to contributing to fields of research that are currently central to the Irish-language community; such as lexicography, language learning, literary analysis, and terminology among others (Knight et al., 2021a, 2021b). Contributions to the fields of Natural Language Processing or Computational Linguistics were identified during the planning phase, but were not at the core of the project. This approach is therefore contrary to the approach taken by a number of data providers whose strengths lie in the scale of their data rather than their knowledge of a language community. The 2301 OSCAR (Open Super-large Crawled Aggregated coRpus) dataset contains 221.81 million words of Irish (Abadji et al., 2022) and is a good example of a dataset that delivers a large number of words from web sources. The aim of the OSCAR project is to collect as much text as possible in a language that has been detected automatically. While they filter out machine-translation content as best as possible, a highly-fluent speaker of Irish would still detect machine-translated content in the OSCAR dataset. This is a task that is difficult to automate for the OSCAR team for lower-resourced languages like Irish, not a criticism of their process. The observation is included here with a view to differentiating our dataset from others.
Other important considerations for the creators of these by-products were that they be created without infringing on copyright (Iacino et al., 2024), and without going against both the terms of use and the trust of the people who shared data with the CNG project. Permission was granted by copyright holders to make the text searchable via a website and, crucially, to generate by-products from the data. This, however, was done as the ChatGPT and general GenAI boom was beginning. Whilst criticism for these technologies was not unanimous within the Irish-language community, a significant cohort of the community gathered to voice their displeasure with AI-generated content and the negative impact these emerging technologies may have on the creative arts and industries. This displeasure was visible on social media and within the community in general in 2024, and culminated with a government minister being drawn to comment on AI use in state organisations (Ó Coimín & Ó Dubhgáin, 2026) and in Údarás na Gaeltachta3 receiving heavy criticism for an AI-based competition they ran in 2025 (Brumell, 2025; Ó Coimín, 2026; Ó Liatháin, 2026). Although permission had been granted, the authors appreciated the shifting landscape and elected not to include any of the following in the dataset of randomised sentences: creative works of any sort, spoken data of any sort, copyrighted data, data belonging to organisations that had not been published on the Web. The data used includes news articles, legal texts, government reports, advertisements, and some social media or blog posts, among other text types.
(2) Method
Python programs were used to generate the datasets from the collected data, we describe key features of each dataset. All data were stored in directories that corresponded to the source from which they were collected. This and metadata served as a step for excluding any content that was not to be included in the three datasets described. The data in these directories were stored as plain text files and encoded as UTF-8. Meta-text such as boiler plate or front and back matter, footnotes or indexes, were removed but no corrections nor spellchecking was conducted. Each file was given a unique alpha-numeric identifier which denoted its source, followed by a six-digit number.
Steps
Randomised sentence dataset. Directories containing the data that satisfied all criteria that were set out in the Context section of this paper were identified and listed. The result of this process was the 76-million words of data, a subset of the 150-million words collected, from which the randomised sentences dataset could be generated. Sentences were detected using the NLTK (Bird, 2006) sentence tokenizer (version 3.9) followed by a set of regular expressions derived from known Irish-language abbreviations (for example: “Uimh.” from “Uimhir”, which means number). The contents of these directories were streamed into one large TXT file, one sentence per line, at which time every line was numerically indexed. The shuffling process was then conducted using Python’s random library (Python Software Foundation, 2026) to change the position of each line within the large index file. The initially assigned index numbers were not updated so that the program could verify lines were not put back alongside one another at a different position in the output document. This criterion was satisfied with a generator that iterated lines of the document and that checked relative indexes as a boolean.
Warnings were printed if any index numbers were found to be adjacent to one another in the final iteration of the dataset. No warnings were produced on the last run of the program. Line indexes were not included in the final version of the dataset file.
The final dataset contains 1,951,584 sentences or 76,578,449 words (counted using the Linux WC utility). The dataset is distributed as a ZIP archive with a total file size of 173.9 MB.
Word embeddings using word2vec and fasttext. All TXT files were processed using the Gensim 4.3.3 library (Řehůřek & Sojka, 2010), first calling their word2vec algorithm and outputting a space-separated file which contained words in order of their frequency, followed by a 100-dimensional vector representing that word’s embedding. The dataset generated using fasttext followed the same procedure and output format, the only difference being the algorithm that was used to generate the values for each word’s 100 dimensions. The word2vec dataset is distributed as a ZIP archive with a total file size of 122.5 MB. The fasttext dataset is distributed as a ZIP archive with a total file size of 121.5 MB. These datasets were generated using the 101 million words of the National Corpus of Irish, after anonymisation and cleaning had been conducted. The vocabulary size is 249,127 in both and they are stored as plain text in a VEC file.
(3) Dataset Description
Repository name
The datasets are available on Zenodo or on the CNG project website, www.corpas.ie or https://www.corpas.ie/en/extras/.
Format names and versions
Datasets are in version 1, and stored as zipped plain text files.
Creation dates
2022-01-01 to 2025-01-30
Publication date
2026-07-06
Dataset creators
Mícheál J. Ó Meachair, Brian Ó Raghallaigh., Michal B. Měchura, Gearóid Ó Cleircín, & Úna Bhreathnach.
Language
Irish (GA-ie)
License
CC BY-SA 4.0
(4) Reuse Potential
Irish is a lower-resourced language in terms of shareable datasets and language technologies. Government of Ireland (2022) set out a list of technologies and areas of research that require attention. Despite being written before the most recent Generative AI boom, after which paradigms have shifted, a number of the recommendations made in this document still require attention and we believe these datasets will help progress research in these areas.
It is hoped that the randomised sentences will contribute to a relatively broad spectrum of NLP tasks for the Irish language, including speech recognition and synthesis, named entity recognition, among others. It is hoped that the word embeddings contribute to semantic-analysis work for the Irish language from both a linguistic and technological perspective.
The decontextualisation of the randomised sentences should be noted, as it may limit reusability for tasks requiring discourse- or document-level context.
Notes
[1] Project paper forthcoming, project website: www.corpas.ie.
[2] Fluent speakers, in this context, refers to members of the language community who can express themselves and engage in discourse or conversation without needing to switch to another language. Judgements were made based on the experience and knowledge of the authors, and did not require adherence to any written standard or particular spoken dialect.
AI Declaration
We, the authors, declare that Generative AI tools were not employed at any stage of the manuscript drafting. Nor were Generative AI tools used for any other tasks including data acquisition or processing. We declare this in accordance with the publisher’s AI policy (https://ubiquitypress.com/ai-policy).
Acknowledgements
We would like to acknowledge the support of the Heads of School (Ciarán Mac an Bhaird & Gearóidín Uí Laighléis) who supported the CNG project from conception to launch. We would also like to acknowledge the Gaois research assistants who have worked on this project. Last, but by no means least, we acknowledge the contribution of the Irish-language community who support and use these resources.
Author Contributions
Mícheál J. Ó Meachair: Writing – original draft, conceptualisation; Brian Ó Raghallaigh., Michal B. Měchura, Gearóid Ó Cleircín, Úna Bhreathnach: Conceptualisation.