(1) Overview
Repository location
https://doi.org/10.5281/zenodo.19458765
GitHub repository: https://github.com/lumenauta/south-american-phylogenies
Context
This dataset was created as part of an MSc dissertation investigating the application of computational methods to historical linguistics, with a particular focus on South American languages. The primary goal of this research was to evaluate the extent to which methods such as automatic cognate detection and phylogenetic inference can contribute to the study of genealogical relationships in a linguistically diverse and underdocumented region.
South America contains a substantial portion of the world’s linguistic diversity, yet its languages remain relatively underrepresented in large-scale quantitative research. One major obstacle is the heterogeneity of available lexical data sources, which often differ in transcription practices, glossing conventions, and even the names used for languages and families. These inconsistencies hinder interoperability and limit the reuse of existing resources.
To address these challenges, the present dataset aggregates lexical material from a wide range of publicly available sources, including contemporary wordlists and reconstructed proto-language forms, and standardizes them into a unified format suitable for computational research in quantitative diachronic linguistics.
(2) Method
Steps
The dataset was constructed through a multi-step process involving data collection, collation and standardization.
First, lexical data were gathered from multiple publicly available repositories and publications. These include large-scale databases such as Lexibank (List et al., 2022), as well as specialized resources like the Languages of Hunter-Gatherers and their Neighbors database (Bowern et al., 2013; Epps, 2013), alongside a variety of published sources for proto-language reconstructions (Bowern et al., 2013; Birchall et al., 2016; Blum et al., 2023; Brid et al., 2022; Chacon, 2017; Chacon et al., 2019; Constenla Umaña, 2005; Ferraz Gerardi & Reichert, 2021; Ferraz Gerardi et al., 2021; Galucio et al., 2015; Koch-Grünberg, 1914; Walker & Ribeiro, 2011; Zariquiey et al., 2023). The dataset therefore combines sources dealing with both contemporary and reconstructed forms.
Second, all collected data were converted into a unified structure conforming to the CLDF Wordlist specification (Forkel et al., 2018). The final dataset consists of linked tabular CSV files for forms, languages, parameters, and bibliographic sources, together with JSON metadata describing table relationships and schema information. Each lexical form is linked to one or more bibliographic source identifiers through a dedicated source field referencing a BibTeX bibliography. Language metadata were separated into a dedicated language table, while concept metadata were stored in a parameter table to reduce redundancy and improve interoperability.
Third, extensive standardization procedures were applied. Language names were mapped to Glottolog identifiers (Hammarström, 2015; Hammarström et al., 2025) to ensure consistency across sources and naming conventions. Concept labels were aligned manually with Concepticon reference concepts (List et al., 2025), enabling cross-dataset comparability. This process involved resolving differences in glossing conventions across sources and selecting the closest available Concepticon concept for each lexical item.
Lexical forms were also normalized where necessary through a combination of Python preprocessing and regular expression filtering. Processing included normalization of Unicode encoding, replacement of inconsistent transcription symbols, harmonization of delimiter conventions, and tokenization of segment sequences. In cases where source forms used non-standard or non-IPA symbols that could be converted unambiguously, regularized IPA representations were adopted.
Finally, the dataset was cleaned and consolidated. Duplicate entries were removed, inconsistent annotations were resolved, and missing or ambiguous metadata values were standardized where possible. Extant and reconstructed proto-language forms were integrated into a single CLDF dataset to facilitate comparative analyses across synchronic and diachronic data.
Sampling strategy
The dataset aims to maximize coverage within South America, subject to data availability. Languages were included based on the availability of lexical data in accessible sources and the feasibility of mapping their entries to standardized concept sets.
The final dataset contains 299 languages, including 284 extant languages and 15 reconstructed proto-languages, representing 33 language lineages (families and isolates). In total, the dataset contains 25,203 lexical forms. Coverage is uneven across families, reflecting differences in documentation availability and the size of source datasets.
The number of lexical entries per language ranges from 1 to 832, with a mean of 84.3 entries per language (SD = 144.9; median = 8). This uneven distribution reflects the aggregation of heterogeneous source materials with differing lexical coverage.
This sampling strategy is therefore best characterized as opportunistic but coverage-oriented, prioritizing breadth and interoperability over strict balance.
Quality control
Quality control focused on ensuring internal consistency through systematic inspection. Data processing relied on a combination of Python scripts, RegEx-based transformations, and manual inspection. Automated processing was primarily used for format conversion, identifier mapping, tokenization, duplicate detection, and consistency checks, while manual review was used to validate problematic or ambiguous cases.
Transcription and gloss normalization were handled through a combination of rule-based and manual cleaning steps, including standardization of inconsistent delimiters and markers (e.g., “/”, “~”, “;”), removal of non-phonological symbols (e.g., hyphens, underscores, angle brackets, numerals, and stray punctuation), and harmonization of apostrophe variants. For instance, the symbol “:” was disambiguated and mapped to the long vowel marker “ː”, as this is a common tool for the ipapy library before being reviewed one by one.
Concepticon mappings received particular attention during manual review due to differences in glossing granularity and semantic scope across sources. Manual inspection was used to resolve ambiguous mappings conservatively and improve cross-dataset comparability.
Cross-source consistency checks were applied to identify duplicate entries and conflicting annotations. Duplicates were removed based on matching language, concept, and form fields, while discrepancies between sources were resolved conservatively through manual inspection where possible.
Finally, the dataset structure was validated against the CLDF specification to ensure structural integrity and interoperability. Validation included verification of table schemas, referential integrity between tables, and bibliographic source references using the cldf validate utility.
(3) Dataset Description
Repository name
Zenodo
Object name
South American Lexical Dataset (CLDF)
Format names and versions
CLDF (Tables in CSV format with JSON metadata)
Creation dates
Start date: 2025-08-07
End date: 2026-05-21
Dataset creators
Álvaro Arozarena Gómez – data collection, curation, standardization, and documentation
Language
Metadata in English; lexical data from multiple South American languages
License
CC BY 4.0
Publication date
2026-05-21
(4) Reuse Potential
This dataset has significant potential for reuse across multiple domains within and beyond diachronic linguistics.
Within historical linguistics, it can serve as input for computational methods such as automatic cognate detection and phylogenetic inference, enabling large-scale comparative analyses across language families, which was its original purpose. It is particularly well suited for testing hypotheses about genealogical relationships, language contact, and areal diffusion in South America.
The dataset is also relevant for typological research, as its standardized structure facilitates cross-linguistic comparisons. Additionally, it may support work in other areas of computational linguistics, including the development and evaluation of algorithms for lexical similarity, clustering, and language classification.
Beyond research applications, the dataset can be used in teaching contexts to illustrate principles of linguistic diversity, language change, and data standardization. Its adherence to CLDF standards also makes it interoperable with other datasets in the Cross-Linguistic Linked Data ecosystem, supporting integration and aggregation.
Several limitations should be noted. Data coverage is uneven across languages and families, reflecting the availability of source materials. The integration of multiple sources introduces variability in data quality and transcription practices. Despite these limitations, the dataset provides a substantial and reusable resource for the study of language change in a region that remains underexplored in quantitative research.
AI Declaration
Generative AI tools were used to assist in drafting and editing the manuscript. All content was reviewed and validated by the author.
Acknowledgements
I would like to thank my supervisors for their guidance and support during the development of this dataset, as well as Fernando O. de Carvalho, for his guidance on reconstructed forms, and the creators of the original data sources used in this work.
Author Contributions
Conceptualization: Álvaro Arozarena Gómez, Annemarie Verkerk, Marc Tanti
Data curation: Álvaro Arozarena Gómez
Formal analysis: Álvaro Arozarena Gómez
Methodology: Álvaro Arozarena Gómez, Annemarie Verkerk, Marc Tanti
Software: Álvaro Arozarena Gómez
Supervision: Annemarie Verkerk, Marc Tanti
Writing – original draft: Álvaro Arozarena Gómez
Writing – review & editing: Annemarie Verkerk, Marc Tanti