Is more always better? Measuring the quality of ranking data through information entropy Cover

.blurhash-client-img { display: none !important; }

Is more always better? Measuring the quality of ranking data through information entropy

Journal of Data and Information Science

Volume 11 (2026): Issue 1 (February 2026)

By: Yishan Liu, Yu Xiao, Xin Long and Jun Wu

Open Access

|Nov 2025

Figures & Tables

Comparison of rank aggregation under different distributions of input ranking information. In (a), the input information is concentrated among a few objects, whereas in (b), the input information is evenly distributed.

Network representation of input information with varying distributions. (a) illustrates the network corresponding to a concentrated input distribution, whereas (b) illustrates the network for an evenly distributed input.

A measurement of input ranking information quality based on degree entropy (Hd): the more evenly the information is distributed, the higher Hd; the more concentrated the information, the lower Hd.

A measurement of input ranking information quality based on edge-weighted entropy (Hw): the more evenly the information is distributed, the higher Hw; the more concentrated the information, the lower Hw.

Degree entropy (Hd) and edge-weighted entropy (Hw) under different distribution parameters (α) with L0=2,3 and 4 respectively, where n=20, m=1,000. The results are averaged over 100 independent trials.

The relative change in Kendall’s tau-b (∆τb) under different distribution parameters (α), where n=50, mb = 1,000, mc = 2,000 and L0 = 10. Different colors represent different data generation models, e.g. green color denotes the MM. The results are averaged over 100 independent trials.

The Kendall’s tau-b (τb) under different distribution parameters (α), where n = 50, m = 1,000, and L0 = 10. The results are averaged over 100 independent trials.

Degree entropy (Hd) under different distribution parameters (α) and numbers of rankings (m), where n = 50 and L0 = 10. The results are averaged over 100 independent trials.

Edge-weighted entropy (Hw) under different distribution parameters (α) and numbers of rankings (m), where n=50 and L0=10. The results are averaged over 100 independent trials.

Computational efficiency of degree entropy (Hd) under different numbers of objects (n) and rankings (m). The results are averaged over 100 independent trials.

Computational efficiency of edge-weighted entropy (Hw) under different numbers of objects (n) and rankings (m). The results are averaged over 100 independent trials.

The collaborative impact of the length of input rankings (L0) and distribution parameters (α). Each cell represents Kendall’s tau-b under different combinations of L0 and α. The gradual change in color from blue to red indicates a gradual increase in Kendall’s tau-b. The results are averaged over 100 independent trials.

Degree entropy (Hd) and edge-weighted (HW) of the course evaluation dataset under different distribution characteristics, with n=9 and m=146_ The results are averaged over 100 independent trials_

Method	m_n
Method	15	30	45	60	75	90	105	120
H_d	2.195	2.191	2.189	2.181	2.177	2.173	2.166	2.158
H_w	3.577	3.566	3.558	3.539	3.527	3.517	3.496	3.473

Spearman’s rho of the data generated by the three ranking data generation models under different values of α_ The results are averaged over 100 independent trials_

Model	α
Model	0.00	1.00	2.00	3.00	4.00	5.00	6.00	7.00	8.00
MM	0.519	0.510	0.522	0.532	0.534	0.462	0.507	0.535	0.496
P-L	0.659	0.661	0.624	0.685	0.669	0.638	0.677	0.660	0.692
IA	0.807	0.817	0.852	0.901	0.895	0.919	0.890	0.908	0.872

Degree entropy of the data generated by the three ranking data generation models under different values of α_ The results are averaged over 100 independent trials_

Model	α
Model	0.00	1.00	2.00	3.00	4.00	5.00	6.00	7.00	8.00
MM	3.912	3.912	3.895	3.812	3.688	3.511	3.351	3.211	3.102
P-L	3.912	3.912	3.895	3.814	3.688	3.515	3.354	3.213	3.113
IA	3.912	3.912	3.895	3.812	3.685	3.510	3.354	3.211	3.106

Degree entropy (Hd) and edge-weighted (Hw) of the election dataset under different distribution characteristics, with n=9 and m=28,245_ The results are averaged over 100 independent trials_

Method	m_n
Method	1,000	2,000	3,000	4,000	5,000	6,000	7,000	8,000
H_d	2.191	2.188	2.183	2.178	2.171	2.156	2.118	2.082
H_w	3.568	3.561	3.548	3.536	3.516	3.478	3.383	3.292

Kendall’s tau of the data generated by the three ranking data generation models under different values of α_ The results are averaged over 100 independent trials_

Model	α
Model	0.00	1.00	2.00	3.00	4.00	5.00	6.00	7.00	8.00
MM	0.393	0.394	0.399	0.439	0.417	0.383	0.396	0.347	0.373
P-L	0.517	0.540	0.497	0.564	0.505	0.519	0.570	0.551	0.592
IA	0.782	0.773	0.781	0.785	0.782	0.775	0.784	0.803	0.795

Edge-weighted entropy of the data generated by the three ranking data generation models under different values of α_ The results are averaged over 100 independent trials_

Model	α
Model	0.00	1.00	2.00	3.00	4.00	5.00	6.00	7.00	8.00
MM	7.098	6.400	5.513	4.964	4.643	4.444	4.319	4.232	4.168
P-L	7.097	6.399	5.511	4.965	4.642	4.446	4.318	4.232	4.168
IA	7.098	6.399	5.512	4.965	4.640	4.445	4.318	4.230	4.168

Kendall’s tau-b between the aggregated rankings obtained by the five methods from three sets of baseline data and the ground truth rankings, where mb = 1,000, n = 50 and L0 = 10_ The results are averaged over 100 independent trials_

Model	BM	DM	VB	MVR	CG
MM	0.89	0.89	0.94	0.93	0.96
P-L	0.92	0.91	0.96	0.94	0.97
IA	0.93	0.92	0.95	0.96	0.98

DOI: https://doi.org/10.2478/jdis-2025-0055 | Journal eISSN: 2543-683X | Journal ISSN: 2096-157X

Journal RSS Feed

Language: English

Page range: 105 - 131

Submitted on: Jul 1, 2025

|

Accepted on: Oct 13, 2025

|

Published on: Nov 7, 2025

Published by: Chinese Academy of Sciences, National Science Library

In partnership with: Paradigm Publishing Services

Keywords:

Multiple criteria decision making,

Rank aggregation,

Information entropy

Related subjects:

Computer sciences,

Information technology,

Computer sciences,

Project management,

Computer sciences,

Databases and data mining

© 2025 Yishan Liu, Yu Xiao, Xin Long, Jun Wu, published by Chinese Academy of Sciences, National Science Library
This work is licensed under the Creative Commons Attribution 4.0 License.

Previous article Volume 11 (2026): Issue 1 (February 2026)Next article