Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Leydesdorff, Loet; Wagner, Caroline S.; Zhang, Lin

doi:10.2478/jdis-2021-0014

Figures & Tables

Potentially overlapping confidence intervals of the PP-Top10% for three leading Chinese universities.

Grouping of significantly different and non-different values for PP-top 10% among three Chinese universities; z-values (provided in the figure) are used for sizing the nodes.

Grouping of 205 Chinese universities in terms of z-values (p <.01); VOSviewer used only for the visualization; modularity Q = 0.165. The figure can be web-started from here (or with a black background from here).

Development of the values of P, P-top10%, and PP-Top10% for Fudan University during 2016–2020.

Grouping of 205 Chinese universities based on overlapping confidence intervals; VOSviewer used for the decomposition and clustering; font- and node-sizes based on z-values. (The map can be web-started from here.)

Grouping of 205 Chinese and 197 US universities in terms of z-values; differences among groups are significant at the 1% level; VOSviewer used for the decomposition and clustering. (The map can be web-started from here.)

Distribution of z values in decreasing order for 205 Chinese and 197 American universities.

The participation of Fudan University in the top 10% class of papers using the Leiden Rankings for subsequent years as a time series versus the reconstruction using the 2020-model.

75 top universities with overlapping confidence values_

University	Z	rank	within group
Tsinghua University	11.005	1	1
Hunan University	10.193	2	2
Hong Kong University of Science and Technology	6.566	3	3
University of Science and Technology of China	6.482	4	4
City University of Hong Kong	6.454	5	5
Shandong University of Science and Technology	6.444	6	6
Hong Kong Polytechnic University	6.406	7	7
South China University of Technology	6.049	8	8
Chinese University of Hong Kong	5.993	9	9
Nankai University	4.860	10	10
University of Hong Kong	4.418	11	11
Shenzhen University	4.089	12	12
Zhejiang University	3.800	13	13
Qufu Normal University	3.724	14	14
Fuzhou University	3.487	15	15
Beihang University	3.432	16	16
University of the Chinese Academy of Sciences	3.354	17	17
Harbin Institute of Technology	3.226	18	18
Central China Normal University	3.090	19	19
University of Macau	3.046	20	20
Northwestern Polytechnical University	2.915	21	21
Wuhan University of Technology	2.914	22	22
Huazhong University of Science and Technology	2.907	23	23
Peking University	2.689	24	24
Southern University of Science and Technology	2.332	25	25
Beijing University of Chemical Technology	2.273	26	26
GuangZhou University	2.120	27	27
Hong Kong Baptist University	2.094	28	28
Nanjing University	1.973	29	29
Xiamen University	1.954	30	30
Wuhan University	1.765	31	31
Tianjin University	1.738	32	32
Dalian University of Technology	1.553	33	33
Shandong Normal University	1.478	34	34
East China Normal University	1.333	35	35
Jiangsu Normal University	1.313	36	36
Soochow University	1.310	37	37
Zhejiang Normal University	1.238	38	38
University of Jinan	1.195	39	39
University of Electronic Science and Technology of China	1.154	40	40
Nanjing University of Science and Technology	1.140	41	41
Nanjing University of Information Science and Technology	1.093	42	42
Southwest Jiaotong University	1.088	43	43
China University of Geosciences	1.035	44	44
Huazhong Agricultural University	1.030	45	45
Tongji University	0.954	46	46
University of Science and Technology Beijing	0.932	47	47
China Agricultural University	0.704	48	48
Beijing Institute of Technology	0.587	49	49
Sun Yat-sen University	0.572	50	50
Hangzhou Dianzi University	0.449	51	51
North China Electric Power University	0.414	52	52
China University of Petroleum Beijing	0.389	53	53
Guangdong University of Technology	0.332	54	54
Shanghai University	0.332	55	55
Fourth Military Medical University	0.257	56	56
Beijing Normal University	0.178	57	57
Chongqing University	0.000	58	58
Hefei University of Technology	0.000	59	59
Xi’an Jiaotong University	0.000	60	60
Nanjing University of Posts and Telecommunications	−0.110	61	61
Heilongjiang University	−0.158	62	62
Nanjing Agricultural University	−0.171	63	63
China Pharmaceutical University	−0.259	64	64
Central South University	−0.280	65	65
China University of Petroleum East China	−0.407	66	66
Qingdao Agricultural University	−0.422	67	67
Beijing Jiaotong University	−0.448	68	68
Chang’an University	−0.465	69	69
Hubei University	−0.498	70	70
Dalian Medical University	−0.643	72	71
Chongqing University of Posts and Telecommunications	−0.656	73	72
Huaqiao University	−0.795	74	73
Anhui Agricultural University	−0.862	75	74
Army Medical University	−0.952	80	75

z-values for nodes and links among three leading Chinese universities_

	Peking University	Tsinghua University	Zhejiang University
Peking University	2.689 **
Tsinghua Univ.	8.460 ***	11.005 ***
Zhejiang University	0.638	8.533 ***	3.800 ***

Standardized residuals of the chi-square values in Table 4_

	Top 10%	Non-top
Tsinghua	5.84	−2.19
Zhejiang	−5.37	2.01

Bottom group of 102 Chinese universities_

University	z	rank	within-group
Hubei University	−0.498	70	1
Northwest University	−1.609	100	2
South China Agricultural University	−1.631	101	3
Zhejiang University of Technology	−1.676	102	4
China University of Mining and Technology	−1.678	103	5
Nanjing University of Aeronautics and Astronautics	−1.816	106	6
Second Military Medical University	−1.844	108	7
Zhejiang Sci-Tech University	−1.886	110	8
Taiyuan University of Technology	−1.934	111	9
Shanghai University of Traditional Chinese Medicine	−1.950	112	10
Qingdao University of Science and Technology	−1.961	113	11
Nanjing University of Chinese Medicine	−1.981	114	12
Harbin Engineering University	−2.033	115	13
Lanzhou University	−2.110	117	14
Northeast Normal University	−2.128	118	15
China Jiliang University	−2.130	119	16
Jiangnan University	−2.163	120	17
Jinan University	−2.227	121	18
Hangzhou Normal University	−2.245	122	19
Southern Medical University	−2.263	123	20
Hunan Normal University	−2.282	124	21
Henan University	−2.328	125	22
Beijing University of Chinese Medicine	−2.346	126	23
Tianjin Medical University	−2.348	127	24
Shantou University	−2.383	128	25
Xi’an University of Architecture and Technology	−2.384	129	26
Hainan University	−2.417	130	27
Hohai University	−2.453	131	28
Beijing University of Posts and Telecommunications	−2.458	132	29
Qingdao University	−2.465	133	30
Jiangsu University	−2.467	134	31
Jiangxi Normal University	−2.486	135	32
Shaanxi Normal University	−2.505	136	33
Shandong University of Technology	−2.506	137	34
Nanchang University	−2.528	138	35
Henan Agricultural University	−2.584	139	36
Guizhou University	−2.675	140	37
Guangxi Normal University	−2.709	141	38
Northeast Forestry University	−2.751	142	39
Northeastern University	−2.761	143	40
Henan Normal University	−2.775	144	41
Shandong Agricultural University	−2.783	145	42
Anhui Normal University	−2.834	146	43
Lanzhou University of Technology	−2.835	147	44
Ningbo University	−2.838	148	45
Southwest University	−2.855	149	46
Yunnan University	−2.940	150	47
Shanghai Ocean University	−2.954	151	48
Northwest Normal University	−2.962	152	49
Inner Mongolia University	−2.968	153	50
Beijing University of Technology	−2.969	154	51
Fudan University	−3.065	155	52
Yanshan University	−3.100	156	53
Tianjin University of Science and Technology	−3.150	157	54
Beijing Forestry University	−3.151	158	55
People's Liberation Army University of Science and Technology	−3.152	159	56
Capital Normal University	−3.188	160	57
Southwest University of Science and Technology	−3.433	161	58
Hebei University of Technology	−3.558	162	59
Shenyang Pharmaceutical University	−3.713	163	60
Xinjiang University	−3.741	164	61
Kunming University of Science and Technology	−3.832	165	62
Nantong University	−3.935	166	63
Xiangtan University	−4.012	167	64
Northeast Agricultural University	−4.025	168	65
Wenzhou Medical University	−4.062	169	66
Chongqing Medical University	−4.106	170	67
Guangxi University	−4.119	171	68
Xi’An University of Technology	−4.203	172	69
Zhengzhou University	−4.244	173	70
Ocean University of China	−4.246	174	71
Henan Polytechnic University	−4.348	175	72
Shihezi University	−4.400	176	73
Kunming Medical University	−4.506	177	74
Xuzhou Medical College	−4.604	178	75
Tianjin Polytechnic University	−4.775	179	76
South China Normal University	−4.827	180	77
Nanjing Medical University	−4.941	181	78
Chengdu University of Technology	−4.958	182	79
Xinxiang Medical University	−4.962	183	80
Harbin University of Science and Technology	−5.001	184	81
North University of China	−5.068	185	82
Yangtze University	−5.093	186	83
Guangzhou Medical University	−5.200	187	84
Sichuan Agricultural University	−5.212	188	85
Sichuan University	−5.247	189	86
National University of Defense Technology	−5.257	190	87
Harbin Medical University	−5.331	191	88
Tianjin Normal University	−5.358	192	89
Shanxi Medical University	−5.378	193	90
Henan University of Science and Technology	−5.416	194	91
Shandong University	−5.619	195	92
Anhui Medical University	−5.951	196	93
Hebei University	−6.307	197	94
Guangxi Medical University	−7.267	198	95
Peking Union Medical College	−7.621	199	96
Xinjiang Medical University	−7.703	200	97
China Medical University	−8.064	201	98
Fujian Medical University	−8.169	202	99
Hebei Medical University	−8.966	203	100
Jilin University	−9.607	204	101
Capital Medical University	−12.944	205	102

Number of research universities included in LR 2020 for respective countries_

Country (top 10 of 65)	N of universities
China	205
United States	198
United Kingdom	58
Germany	54
Japan	53
South Korea	44
Italy	41
Spain	41
India	36
Iran	36	%

sum	766	65.1%

<55 other countries>	= (1,176 – 766) = 410	34.9%

65 countries	1,176	100%

Top group of 32 Chinese universities_

University	Z	overall rank	within-group rank
Tsinghua University	11.005	1	1
Hunan University	10.193	2	2
Hong Kong University of Science and Technology	6.566	3	3
University of Science and Technology of China	6.482	4	4
City University of Hong Kong	6.454	5	5
Shandong University of Science and Technology	6.444	6	6
Hong Kong Polytechnic University	6.406	7	7
South China University of Technology	6.049	8	8
Chinese University of Hong Kong	5.993	9	9
Nankai University	4.860	10	10
University of Hong Kong	4.418	11	11
Shenzhen University	4.089	12	12
Qufu Normal University	3.724	14	13
Fuzhou University	3.487	15	14
Beihang University	3.432	16	15
University of the Chinese Academy of Sciences	3.354	17	16
Central China Normal University	3.090	19	17
University of Macau	3.046	20	18
Northwestern Polytechnical University	2.915	21	19
Wuhan University of Technology	2.914	22	20
Southern University of Science and Technology	2.332	25	21
Beijing University of Chemical Technology	2.273	26	22
GuangZhou University	2.120	27	23
Hong Kong Baptist University	2.094	28	24
Shandong Normal University	1.478	34	25
Jiangsu Normal University	1.313	36	26
Zhejiang Normal University	1.238	38	27
University of Jinan	1.195	39	28
Nanjing University of Information Science and Technology	1.093	42	29
Hangzhou Dianzi University	0.449	51	30
Heilongjiang University	−0.158	62	31
Qingdao Agricultural University	−0.422	67	32

Chi-square for the comparison of Tsinghua and Zhejiang University in LR 2020_

Chi-square	top 10%	non-top
Tsinghua	34.10	4.79	38.89
Zhejiang	28.87	4.05	32.92
		χ² =	71.80

Middle group of 69 Chinese universities_

University	z	overall rank	within-group
Zhejiang University	3.800	13	1
Harbin Institute of Technology	3.226	18	2
Huazhong University of Science and Technology	2.907	23	3
Peking University	2.689	24	4
Nanjing University	1.973	29	5
Xiamen University	1.954	30	6
Wuhan University	1.765	31	7
Tianjin University	1.738	32	8
Dalian University of Technology	1.553	33	9
East China Normal University	1.333	35	10
Soochow University	1.310	37	11
University of Electronic Science and Technology of China	1.154	40	12
Nanjing University of Science and Technology	1.140	41	13
Southwest Jiaotong University	1.088	43	14
China University of Geosciences	1.035	44	15
Huazhong Agricultural University	1.030	45	16
Tongji University	0.954	46	17
University of Science and Technology Beijing	0.932	47	18
China Agricultural University	0.704	48	19
Beijing Institute of Technology	0.587	49	20
Sun Yat-sen University	0.572	50	21
North China Electric Power University	0.414	52	22
China University of Petroleum Beijing	0.389	53	23
Guangdong University of Technology	0.332	54	24
Shanghai University	0.332	55	25
Fourth Military Medical University	0.257	56	26
Beijing Normal University	0.178	57	27
Chongqing University	0.000	58	28
Hefei University of Technology	0.000	59	29
Xi’an Jiaotong University	0.000	60	30
Nanjing University of Posts and Telecommunications	−0.110	61	31
Nanjing Agricultural University	−0.171	63	32
China Pharmaceutical University	−0.259	64	33
Central South University	−0.280	65	34
China University of Petroleum East China	−0.407	66	35
Beijing Jiaotong University	−0.448	68	36
Xidian University	−0.529	71	37
Dalian Medical University	−0.643	72	38
Chongqing University of Posts and Telecommunications	−0.656	73	39
Huaqiao University	−0.795	74	40
Anhui Agricultural University	−0.862	75	41
Wuhan University of Science and Technology	−0.899	76	42
Nanjing Tech University	−0.919	77	43
Donghua University	−0.923	78	44
East China University of Science and Technology	−0.943	79	45
Army Medical University	−0.952	80	46
Anhui University	−1.004	81	47
Tianjin University of Technology	−1.047	82	48
Renmin University of China	−1.064	83	49
Nanjing Normal University	−1.189	84	50
Southeast University	−1.213	85	51
Fujian Agriculture and Forestry University	−1.218	86	52
Northwest Agriculture and Forestry University	−1.226	87	53
University of Shanghai for Science and Technology	−1.232	88	54
North China University of Science and Technology	−1.249	89	55
Wenzhou University	−1.306	90	56
Southwest Petroleum University	−1.311	91	57
Shaanxi University of Science and Technology	−1.314	92	58
China Three Gorges University	−1.333	93	59
University of South China	−1.345	94	60
Shanghai Normal University	−1.369	95	61
Dalian Maritime University	−1.380	96	62
Yangzhou University	−1.392	97	63
Changzhou University	−1.544	98	64
Nanjing Forestry University	−1.581	99	65
Guangzhou University of Chinese Medicine	−1.685	104	66
Fujian Normal University	−1.781	105	67
China Academy of Chinese Medical Sciences	−1.840	107	68
Shanghai Jiao Tong University	−1.876	109	69

Expected values of top 10% cited papers for Tsinghua and Zhejiang University_

Expected values	top 10%	non-top	total p
Tsinghua	2,449.01	17,452.99	19,902.00
Zhejiang	2,892.99	20,617.01	23,510.00
	5,356.09	38,055.91	43,412.00

Observed values of top 10% cited papers for Tsinghua and Zhejiang University during the period 2015–2019 (fractional counting, all sciences)_

Observed values	top 10%	non-top	total p
Tsinghua	2,738	17,164	19,902
Zhejiang	2,604	20,906	23,510
	5,342	38,070	43,412

Top-20 universities in each of the three clusters_ (See for a full list in the Annexes_)

	top group (top 20)	z	middle group (top 20)	z	bottom group (top 20)	z
1	Tsinghua Univ.	11.005	Zhejiang Univ.	3.800	Hubei Univ.	−0.498
2	Hunan Univ.	10.193	Harbin Institute of Technology	3.226	Northwest Univ.	−1.609
3	Hong Kong Univ. of Science and Technology	6.566	Huazhong Univ. of Science and Technology	2.907	South China Agricultural Univ.	−1.631
4	Univ. of Science and Technology of China	6.482	Peking Univ.	2.689	Zhejiang Univ. of Technology	−1.676
5	City Univ. of Hong Kong	6.454	Nanjing Univ.	1.973	China Univ. of Mining and Technology	−1.678
6	Shandong Univ. of Science and Technology	6.444	Xiamen Univ.	1.954	Nanjing Univ. of Aeronautics and Astronautics	−1.816
7	Hong Kong Polytechnic Univ.	6.406	Wuhan Univ.	1.765	Second Military Medical Univ.	−1.844
8	South China Univ. of Technology	6.049	Tianjin Univ.	1.738	Zhejiang Sci-Tech Univ.	−1.886
9	Chinese Univ. of Hong Kong	5.993	Dalian Univ. of Technology	1.553	Taiyuan Univ. of Technology	−1.934
10	Nankai Univ.	4.860	East China Normal Univ.	1.333	Shanghai Univ. of Traditional Chinese Medicine	−1.950
11	Univ. of Hong Kong	4.418	Soochow Univ.	1.310	Qingdao Univ. of Science and Technology	−1.961
12	Shenzhen Univ.	4.089	Univ. of Electronic Science and Technology of China	1.154	Nanjing Univ. of Chinese Medicine	−1.981
13	Qufu Normal Univ.	3.724	Nanjing Univ. of Science and Technology	1.140	Harbin Engineering Univ.	−2.033
14	Fuzhou Univ.	3.487	Southwest Jiaotong Univ.	1.088	Lanzhou Univ.	−2.110
15	Beihang Univ.	3.432	China Univ. of Geosciences	1.035	Northeast Normal Univ.	−2.128
16	Univ. of the Chinese Academy of Sciences	3.354	Huazhong Agricultural Univ.	1.030	China Jiliang Univ.	−2.130
17	Central China Normal Univ.	3.090	Tongji Univ.	0.954	Jiangnan Univ.	−2.163
18	Univ. of Macau	3.046	Univ. of Science and Technology Beijing	0.932	Jinan Univ.	−2.227
19	Northwestern Polytechnical Univ.	2.915	China Agricultural Univ.	0.704	Hangzhou Normal Univ.	−2.245
20	Wuhan Univ. of Technology	2.914	Beijing Institute of Technology	0.587	Southern Medical Univ.	−2.263

Descriptive statistics of the comparison among 197 American and 205 Chinese universities_

Row Labels	low	middle	High	Isolates	Grand Total
China	116	67	21	1	205
USA	36	60	99	2	197
Grand Total	152	127	120		402

Highest ranked universities in top-groups in the combined American-Chinese set of 402 universities in LR 2020 with their respective z-scores_

rank	Top 20 Chinese universities	z	Top 20 American Universities	z
1	Tsinghua Univ.	11.005	Harvard Univ.	36.632
2	Hunan Univ.	10.193	Stanford Univ.	26.028
3	Hong Kong Univ. of Science and Technology	6.566	Massachusetts Institute of Technology	24.504
4	Univ. of Science and Technology of China	6.482	Univ. of California, Berkeley	20.319
5	City Univ. of Hong Kong	6.454	Yale Univ.	16.576
6	Shandong Univ. of Science and Technology	6.444	Univ. of California, San Francisco	16.524
7	Hong Kong Polytechnic Univ.	6.406	Princeton Univ.	16.522
8	South China Univ. of Technology	6.049	Columbia Univ.	16.348
9	Chinese Univ. of Hong Kong	5.993	Univ. of California, San Diego	16.061
10	Nankai Univ.	4.860	Univ. of Pennsylvania	15.975
11	Univ. of Hong Kong	4.418	Cornell Univ.	14.934
12	Shenzhen Univ.	4.089	Univ. of Washington, Seattle	14.620
13	Qufu Normal Univ.	3.724	Johns Hopkins Univ.	14.540
14	Fuzhou Univ.	3.487	Northwestern Univ.	14.488
15	Central China Normal Univ.	3.090	Univ. of California, Los Angeles	14.450
16	Univ. of Macau	3.046	Univ. of Michigan	13.951
17	Wuhan Univ. of Technology	2.914	California Institute of Technology	13.695
18	Southern Univ. of Science and Technology	2.332	Univ. of Chicago	13.561
19	GuangZhou Univ.	2.120	Duke Univ.	12.605
20	Hong Kong Baptist Univ.	2.094	Washington Univ. in St. Louis	12.123

Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Figures & Tables

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

75 top universities with overlapping confidence values_

z-values for nodes and links among three leading Chinese universities_

Standardized residuals of the chi-square values in Table 4_

Bottom group of 102 Chinese universities_

Number of research universities included in LR 2020 for respective countries_

Top group of 32 Chinese universities_

Chi-square for the comparison of Tsinghua and Zhejiang University in LR 2020_

Middle group of 69 Chinese universities_

Expected values of top 10% cited papers for Tsinghua and Zhejiang University_

Observed values of top 10% cited papers for Tsinghua and Zhejiang University during the period 2015–2019 (fractional counting, all sciences)_

Top-20 universities in each of the three clusters_ (See for a full list in the Annexes_)

Descriptive statistics of the comparison among 197 American and 205 Chinese universities_

Highest ranked universities in top-groups in the combined American-Chinese set of 402 universities in LR 2020 with their respective z-scores_

Paradigm

My account