Noor-Sharaye v.1. A Benchmark Dataset of Complex Words for Evaluating Arabic Analyzers

Repository location

https://zenodo.org/records/21481222

Context

The dataset was created within a doctoral research effort aimed at advancing computational resources for Classical Arabic morphological processing. It addresses the limited availability of large-scale, high-quality datasets suitable for stemming and morphological analysis of Classical Arabic texts.

It was developed as part of an academic project conducted at the Computer Research Center for Islamic Sciences (Noor) and the School of Computer Engineering at Iran University of Science and Technology (IUST), under the supervision of Prof. Behrouz Minaei-Bidgoli.

The dataset has been used within the scope of a PhD dissertation focusing on Arabic morphological analysis tasks such as stemming, root extraction, and word segmentation.

2.1 Existing Datasets

To evaluate the performance of any model, we utilize existing datasets for Arabic stemming. Table 1 represents some of them.

2.2 existing Arabic stemmers

In the following section, we present and evaluate a set of widely used Arabic stemming approaches within a unified experimental framework. Table 2 shows that.

2.3 Existing Arabic resources

Several Arabic NLP resources and analyzers have been developed to support morphological analysis and text processing. Table 3 compares the dataset with some widely used Arabic resources.

Unlike existing Arabic NLP tools, the dataset provides a manually validated Classical Arabic corpus for morphological analysis, benchmarking, and model evaluation.

Steps

1. Text Preparation and Normalization

The source Classical Arabic texts were first digitized and carefully prepared for annotation. All preprocessing steps, including orthographic normalization and unification of Arabic character variants, were conducted manually under linguistic supervision to preserve textual integrity. The text was then segmented into individual word units while maintaining contextual and morphological boundaries. This manual preparation ensured that the dataset remained faithful to the original linguistic structure of Classical Arabic.

2. Human Morphological Annotation Scheme

Each word in the dataset was manually annotated by trained linguistic researchers using a structured morphological schema. Every token was assigned a rich set of linguistic attributes, including:

• Surface word form

• Lemma

• Part-of-speech (POS) tag

• Grammatical case

• Affix and clitic segmentation (prefixes and suffixes)

• Additional morpho-syntactic features where applicable

All annotations were performed manually without the use of external morphological analyzers, ensuring that the dataset reflects expert linguistic judgment rather than automated predictions.

3. Data Representation

The dataset is provided in multiple formats to support different usage scenarios. The data is available in XLS, JSON, and XML formats. Figure 1 shows that.

Figure 1

3.1 Dataset Schema

Table 4 summarizes the main fields included in the dataset and their linguistic descriptions.

A sample of the morphological annotations included in the dataset is presented in Table 5.

Sampling Strategy

The released dataset consists of 6,838 manually annotated and validated word instances extracted from the larger Noor-Sharaye corpus of approximately 205,000 word instances. The released subset was extracted from Surah Al-Fatihah and Surah Al-Baqarah due to their complementary linguistic characteristics. This combination ensures a balanced and representative sample of Classical Arabic, capturing a wide range of word formation, affixation, and grammatical structures. The resulting subset is suitable for benchmarking and reproducible experimental evaluation, as illustrated in Figure 2.

Figure 2

Quality control

A quality control procedure was applied to ensure the accuracy and consistency of the Noor-Sharaye dataset. All entries were manually reviewed by trained linguists specializing in Classical Arabic morphology, and consistency checks were performed to verify morphological annotations. To assess annotation reliability, three trained linguists independently annotated a random sample of 500 tokens. Inter-annotator agreement was measured using token-level F1-score across five annotation tasks: POS tagging, root extraction, lemma assignment, affix segmentation, and morphological features. The annotation process achieved an overall F1-score of 97.54%, indicating high annotation reliability. Disagreements were resolved through consensus discussion and consultation with a senior linguist.

Repository name

Zenodo

Object name

Noor- Sharaye dataset.

Format names and versions

xls, xml, and json.

Creation dates:

2/3/2021 to 1/7/2022.

Dataset creators

This dataset was developed by a research team working in the field of Arabic natural language processing. Computer Research Center for Islamic Sciences (Noor) data preparation and linguistic support, Behrouz Minaei-Bidgoli provided academic supervision, while Azal Alaswaad conducted dataset analysis, preprocessing, development, and validation.

Language

Arabic

License

Creative Commons Attribution 4.0 International.

Publication date

6-7-2026

Source Documentation

The Noor-Sharaye corpus contains 205,000 morphologically annotated words from 17 Classical Arabic books. A validated subset of 6,838 entries is publicly available via Zenodo. The full corpus is maintained by the Noor Center and is available upon reasonable request.

Representativeness of the Classical Arabic Domain

The dataset is grounded in Classical Arabic as represented across 17 diverse textual sources, including the Quran, Fiqh, Hadith, and historical texts. This diversity ensures comprehensive coverage of linguistic variation across different genres and periods of Classical Arabic. These texts are characterized by:

• Highly structured and systematic lexical formation

• Frequent use of morphological variation and derivation

• Rich inflectional and root-based word formations

The Classical Arabic domain holds exceptional religious, linguistic, and scholarly significance. As the foundation of Arabic linguistic tradition, these texts have been central to grammatical theory, morphological analysis, and lexical studies for centuries. Transforming these texts into a structured computational dataset enables several key contributions:

• Supporting computational analysis of Classical Arabic morphology

• Enabling the development and evaluation of Arabic stemming and deep learning models

• Providing a benchmark resource for sequence-to-sequence and neural architectures

• Facilitating comparative studies between rule-based and data-driven approaches

• Enhancing reproducibility in Arabic NLP research through a standardized gold dataset

5.1 Training AI Models

The dataset can be used as a training corpus for sequence-to-sequence and transformer-based Arabic NLP models. A sequence-to-sequence deep learning architecture combining Bi-LSTM and Transformer models was adopted to model the relationship between input word forms and their corresponding stems, allowing the system to effectively learn and generalize complex morphological structures in Classical Arabic. Figure 3 illustrates an example Python script for loading the dataset.

Figure 3

5.2 Benchmarking and Comparison

We utilized the dataset to perform a systematic comparison between several widely used Arabic stemming approaches, such as Al-Khalil, Light10, PStemmer, NLTK stemmer, Light8, CAMeL, Khoja, and Tashaphyne. This evaluation was conducted as part of our research work, where the dataset served as a consistent benchmark for assessing and comparing different stemming techniques under a unified experimental setting (Al-Aswad et al., 2024).