Enhancing Command Recognition in Air Traffic Control Through Advanced Classification Techniques

Srinivasan, Narayanan; Balasundaram, S. R.

doi:10.2478/tar-2024-0011

Figures & Tables

Algorithms performance for Rest of Callsign

Comparison of classification algorithms for callsign

Comparison of classification algorithms for rest of callsign

Naïve Bayes Classification Results for CallSign

	precision	recall	f1-score	support
CALLSIGN	0.72	1.00	0.84	34
NOCALLSIGN	0.00	0.00	0.00	13
Accuracy			0.72	47
Macro avg	0.36	0.5	0.42	47
Weighted avg	0.52	0.72	0.61	47

Logistic Regression Classification Results for Callsign and NoCallSign

	precision	recall	f1-score	support
CALLSIGN	0.75	0.88	0.81	34
NOCALLSIGN	0.43	0.23	0.30	13
Accuracy			0.70	47
Macro avg	0.59	0.56	0.56	47
Weighted avg	.66	.70	.67	47

Logistic Regression Rest of Callsign Results

	precision	recall	f1-score	support
CLIMB	1	0.62	0.77	8
CONTACT	1	1	1	4
DESCEND	1	1	1	3
DIRECTTO	1	0.5	0.67	4
PROCEEDDIRECT	0.8	1	0.89	4
PROCEED	1	1	1	1
RHEIN	1	0.86	0.92	7
RHEINRADAR	0.94	1	0.97	15
SETCOURSE	0.43	1	0.6	3
SET	0	0	0	0
TURN	0.8	0.67	0.73	6
TURNHEADING	0	0	0	2
TURNLEFT	0.71	1	0.83	5
TURNRIGHT	0.8	1	0.89	4

micro avg	0.85	0.85	0.85	66
macro avg	0.75	0.76	0.73	66
weighted avg	0.87	0.85	0.84	66

LSVM Rest of Callsign Samples

DESCEND FLIGHT TWO ZERO EXPEDITE DESCENT AS CROSSING TRAFFIC	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE EIGHT ZERO	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE FO= SEVEN	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE HUNDRED RATE ONE THOUSAND FIVE HUNDRED OR MORE	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE ONE TWO FIVE	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE ONE TWO	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE ONE ZERO	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE TWO ZERO RATE FIFTEEN HUNDRED OR MORE	[“DESCEND”]
DESCEND FLIGHT LEVEL THREE TWO ZERO	[“DESCEND”]
DESCEND FLIGHT LEVEL TO YOU ZERO SET COURSE TO FOXTROT	[“DESCEND”]

Logistic Regression “Rest of CallSign” Samples

CLIMB FLIGHT LEVEL THREE HUNDRED SET COURSE	[“CLIMB”]
CLIMB FLIGHT LEVEL TWO NINE ALSO SET COURSE TRASADINGEN	[“CLIMB”]
CONTACT ~M YOU ONE ONE NATTENHEIM ONE ONE OF TWO	[“CONTACT”]
CONTACT ONE ONE ~M ONE AND	[“CONTACT”]
CONTACT RHEIN ONE THREE TWO DECIMAL FOUR ~S	[“CONTACT”]
DESCEND FLIGHT SET EIGHT ONE SIX LEFT	[“DESCEND”]
DIRECT TO FOXTROT ROMEO INDIA	[“DIRECTTO”]
RHEIN ~M MINUTE AND ~M ONE ONE ONE	[“RHEIN”]
RHEIN AH AND YOU ONE AH GUTEN	[“RHEIN”]
RHEIN AIR ONE THREE TWO DECIMAL FOUR	[“RHEIN”]
RHEIN ONE AND CALL	[“RHEIN”]

Comparison of Callsign, No Callsign

Algorithm	Callsign Detection	No Callsign Detection
Logistic Regression	63%	86%
LSVM	49%	87%
Naïve Bayes	43%	91%

Summary of results comparison

Work

Technique

Command Error

Recognition Rate

Helmke et al. [4]

SLM+Rescoring (N-best=5)

16.5

Helmke et al. [15]

ASR-SSA-C Concept

35.9

This work

Syntactic Separation for Callsign and Rest of Callsign and using ML classification algorithms

16% (callsign best)

9% (rest of callsign best)

Comparison of Algorithms Callsign and NoCallSIgn

Callsign	No Callsign

LSVM Results Sample

AERO LLOYD FIVE FIVE FIVE	[“CALLSIGN”]
AERO LLOYD FIVE FIVE NINE	[“CALLSIGN”]
AERO LLOYD FIVE FIVE ONE	[“CALLSIGN”]
AERO LLOYD FIVE FIVE SEVEN	[“CALLSIGN”]
AERO LLOYD FIVE FIVE TWO	[“CALLSIGN”]
AERO LLOYD LLOYD	[“NOCALLSIGN”]
AERO LLOYD ONE ONE FOUR	[“CALLSIGN”]
AERO LLOYD TRY A TWO	[“CALLSIGN”]
AERO LLOYD TWO TWO SIX	[“CALLSIGN”]

LSVM “Rest of Callsign” Samples

CLIMB FLIGHT LEVEL THREE FIVE SINGA SET COURSE TO COURSE TO	[“SETCOURSE”]
CLIMB FLIGHT LEVEL THREE HUNDRED SET COURSE	[“SETCOURSE”]
CLIMB FLIGHT LEVEL TWO NINE ALSO SET COURSE TRASADINGEN	[“SETCOURSE”]

Naive Bayes Rest of Callsign Samples

RHEIN IDENT YOU READ	[“RHEIN”]
RHEIN IDENTIFIED CLIMB FLIGHT LEVEL THREE YOU ARE YOU STILL	[“RHEINRADAR”]
RHEIN IDENTIFIED CLIMB FLIGHT LEVEL TWO NINE ALSO SET COSTA THANK YOU	[“RHEINRADAR”]
RHEIN IDENTIFIED CLIMB FLIGHT LEVEL TWO NINE ALSO	[“RHEINRADAR”]
RHEIN IDENTIFIED CLIMB FLIGHT ONE IDENTIFIED	[“RHEINRADAR”]
RHEIN IDENTIFIED CLIMB FLIGHT TO TWO SEVEN ZERO	[“RHEINRADAR”]

Logistic Regression Classification Samples

HEADING FLIGHT HUNDRED MILANO ONE ONE THREE FOUR DECIMAL FIVE TWO	[“NOCALLSIGN”]
HEADING OLYMPIC ONE ONE	[“NOCALLSIGN”]
HEADING ONE ONE ONE ONE	[“NOCALLSIGN”]
HEADING TWO TWO SIX FOUR	[“NOCALLSIGN”]
HEADING YOU CHARLIE PAPA PAPA	[“NOCALLSIGN”]
HIGHER HIGHER	[“NOCALLSIGN”]
HIGHER ONE ONE ONE ONE	[“NOCALLSIGN”]
HOTEL FIVE FIVE SINGA TWO	[“CALLSIGN”]
HOTEL HOTEL	[“CALLSIGN”]

Rest of Callsign – Analysis

Algorithm	Multi Classifier Accuracy	Chart Representation
Logistic Regression	84%
LSVM	83%
Naïve Bayes	58%

Naïve Bayes “Rest of Callsign” Results

	precision	recall	f1-score	support
CLIMB	0.8	1	0.89	8
CONTACT	0.57	1	0.73	4
DESCEND	1	0.33	0.5	3
DIRECTTO	0	0	0	4
PROCEEDDIRECT	0	0	0	4
PROCEED	0	0	0	1
RHEIN	0	0	0	7
RHEINRADAR	0.67	0.93	0.78	15
SETCOURSE	0.25	1	0.4	3
SET	0	0	0	0
TURN	0	0	0	6
TURNHEADING	0	0	0	2
TURNLEFT	0.4	0.8	0.53	5
TURNRIGHT	0.8	1	0.89	4

Micro avg	0.58	0.58	0.58	66
Macro avg	0.32	0.43	0.34	66
Weighted avg	0.42	0.58	0.46	66

LSVM Rest of Callsign Results

	precision	recall	f1-score	support
CLIMB	1	0.62	0.77	8
CONTACT	1	1	1	4
DESCEND	1	1	1	3
DIRECTTO	1	0.25	0.4	4
PROCEEDDIRECT	0.8	1	0.89	4
PROCEED	1	1	1	1
RHEIN	1	0.86	0.92	7
RHEINRADAR	0.94	1	0.97	15
SETCOURSE	0.38	1	0.55	3
SET	0	0	0	0
TURN	0.8	0.67	0.73	6
TURNHEADING	0	0	0	2
TURNLEFT	0.83	1	0.91	5
TURNRIGHT	0.67	1	0.8	4

micro avg	0.83	0.83	0.83	66
macro avg	0.74	0.74	0.71	66
weighted avg	0.86	0.83	0.82	66

LSVM Classification Results for Callsign and No Callsign

	precision	recall	f1-score	support
CALLSIGN	0.75	0.97	0.85	34
NOCALLSIGN	0.67	0.15	0.25	13
Accuracy			0.74	37
Macro avg	0.71	0.56	0.55	47
Weighted avg	0.73	0.74	0.68	47

LSVM No Callsign samples

FOUR FOUR SINGA TWO	[“NOCALLSIGN”]
FOUR FOUR	[“NOCALLSIGN”]
FOX JET JET	[“NOCALLSIGN”]
FOX READ FIVE	[“NOCALLSIGN”]
FOX SWISSAIR SWISSAIR	[“NOCALLSIGN”]
FOXTROT CHECK RADIO CHECK JET	[“NOCALLSIGN”]