A holistic evaluation framework for Chinese graded reading system

Li, Rende; Zhang, Jian

doi:10.2478/jdis-2025-0045

Figures & Tables

The graded reading evaluation criteria system. “Cost” and “Benefit” represent the direction of evaluation for each criterion. “Cost” refers to criteria where a lower value denotes easier reading difficulty. “Benefit” refers to criteria where a higher value denotes easier reading difficulty.

The evaluation framework diagram. The process involves three steps from left to right: collect data, determine criterion weights, and calculate advantage degrees. Firstly, data is collected and aggregated using Probabilistic Fuzzy Linguistic Term (PFLT) and Probabilistic Linguistic Averaging (PLA). Qualitative data is assessed by experts using linguistic terms at five levels: Very Low (VL), Low (L), Medium (M), High (H), and Very High (VH), while quantitative data is directly measured values. Next, triangular and traditional entropy methods are used to calculate weights for the two criteria. Finally, the TODIM method calculates the advantage degree of each book based on the criterion and combines degree to rank the books.

The comparison of book rankings across various evaluation methods. Different colored circles denote different books (B1, B2, …, B12). The vertical axis represents the ranking positions, while the horizontal axis represents the different evaluation methods. The lines of the same style track the changes in rankings for each book, highlighting the trends and variations in their ranking.

The Kendall correlation between different ranking methods: WSTF’s Rank, Lexile’s Rank, Our Rank, Guide’s Rank, Exchange’s Rank, and Remove’s Rank. Significance levels are indicated as follows: *** p < 0.001 (two-tailed), ** p < 0.01 (two-tailed), and * p < 0.05 (two-tailed).

The comparison of ranking under different decision methods.

The comprehensive dominance, η(Bi) and ranking of all candidate books_

Benchmark books	Comprehensive dominance	η(Bi)	Ranking
B1	6.714	1.000	1^th
B2	2.325	0.953	3^th
B3	5.401	0.986	2^th
B4	-14.815	0.769	5^th
B5	-11.959	0.799	4^th
B6	-22.123	0.690	6^th
B7	-37.006	0.530	7^th
B8	-41.461	0.482	9^th
B9	-40.341	0.494	8^th
B10	-75.107	0.121	10^th
B11	-81.314	0.054	11^th
B12	-86.359	0.000	12^th

Linguistic complexity and reading experience of the four Chinese classic novels_

Book title	Linguistic complexity	Reading experience	Reasoning
Journey to the West (西游记)	High difficulty	Low difficulty	Linguistic complexity: Rich in classical Chinese expressions, cultural references, and metaphors.
Journey to the West (西游记)	High difficulty	Low difficulty	Reading experience: Linear and episodic storyline, vivid characters, and engaging plot make it easy to follow and enjoyable.
Romance of the Three Kingdoms (三国演义)	High difficulty	High difficulty	Linguistic complexity: Complex sentence structures, historical terms, and strategic descriptions.
Romance of the Three Kingdoms (三国演义)	High difficulty	High difficulty	Reading experience: Dense historical and military content, with intricate relationships and strategies that require significant understanding.
Water Margin (水浒传)	Medium difficulty	Medium difficulty	Linguistic complexity: More straightforward classical Chinese with less challenging syntax.
Water Margin (水浒传)	Medium difficulty	Medium difficulty	Reading experience: Many characters and subplots demand attention, but the heroic themes and action sequences are engaging.
Dream of the Red Chamber(红楼梦)	High difficulty	High difficulty	Linguistic complexity: Elaborate classical Chinese with poetic and symbolic elements.
Dream of the Red Chamber(红楼梦)	High difficulty	High difficulty	Reading experience: Complex emotional depth, subtle cultural references, and numerous characters and relationships make it demanding.

The entropy value and weight of each criterion_

Criteria	Quantitative criterion
Criteria	C11	C12	C13	C21	C22	C31	C32	C41
Entropy value	0.579	0.602	0.602	0.684	0.730	0.693	0.670	0.728
Initial weight	0.127	0.133	0.132	0.150	0.160	0.152	0.147	0.131
Normalized weights	0.076	0.080	0.079	0.090	0.096	0.091	0.088	0.072
Ranking	8^th	6^th	7^th	3^th	1^th	2^th	4^th	10^th

The existing graded reading systems_

Graded reading system	Description	Limitation
A-Z System (Hiebert & Tortorelli, 2022; McNamara et al., 2014)	Categorizes books into 26 levels (A-Z), covering language difficulty and thematic content, with added factors like font and illustrations.	Primarily designed for English; not easily adaptable to languages with different script systems.
Oxford Reading System (Gorard & See, 2016; Smith & Doe, 2018)	Developed by Oxford University Press, uses vertical levels (based on age, cognitive and emotional development) and horizontal stages (e.g. phonics, comprehension).	Structured for English-speaking readers; lacks accommodation for cultural and linguistic differences in other regions.
Developmental Reading Assessment (DRA) (Beaver & Carter, 2024; Johnson & Lee, 2017)	A U.S. standard assessment evaluating reading comprehension, lexical knowledge, and reading strategies through progressive testing.	Primarily assesses English skills; limited in flexibility for application to other linguistic contexts.
Lexile System (Hiebert, 2005; McNamara et al., 2014; Smith et al., 2016; Zeng & Fan, 2017)	Uses semantic and grammatical complexity to determine reader levels and text difficulty, matching readers with suitable texts.	Focuses on English text; lacks cultural adaptation for non-English readers.
Chinese Southern Graded Reading Center (Nur, 2019; Qiang et al., 2020)	Based on the Lexile framework, divides grades 1-9 into four stages, considering text difficulty, narrative structure, and the integration of text and visuals.	Primarily focuses on linguistic complexity; limited attention to reader interest and emotional engagement.
Shanghai Graded Reading Ability Standards (Holzknecht et al., 2022; Kidwai et al., 2016; Zhao, 2020)	Adapts Lexile’s approach to measure reading attitudes, cognitive processes, and text difficulty in a Chinese context.	Relies on linguistic complexity and overlooks personalized reading interests and emotional dimensions.

A holistic evaluation framework for Chinese graded reading system

Figures & Tables

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

The comprehensive dominance, η(Bi) and ranking of all candidate books_

Linguistic complexity and reading experience of the four Chinese classic novels_

The entropy value and weight of each criterion_

The existing graded reading systems_

Paradigm

My account