Have a personal or library account? Click to login
A Holistic review and performance evaluation of unsupervised learning methods for network anomaly detection Cover

A Holistic review and performance evaluation of unsupervised learning methods for network anomaly detection

International Journal on Smart Sensing and Intelligent Systems
Volume 17 (2024): Issue 1 (January 2024)
By: Niharika Sharma,  Bhavna Arora,  Shabana Ziyad,  Pradeep Kumar Singh and  Yashwant Singh  
Open Access
|May 2024

Figures & Tables

Figure 1:

Bibliometric analysis of AD literature (A) Chronological distribution of bibliometric papers (B) Percentage distribution of publication types. AD, anomaly detection.
Bibliometric analysis of AD literature (A) Chronological distribution of bibliometric papers (B) Percentage distribution of publication types. AD, anomaly detection.

Figure 2:

Roadmap of the paper.
Roadmap of the paper.

Figure 3:

Taxonomy of 13 selected UL methods for anomaly detection from 6 families. UL, unsupervised learning.
Taxonomy of 13 selected UL methods for anomaly detection from 6 families. UL, unsupervised learning.

Figure 4:

Framework of NAD based on UL [45]. NAD, network anomaly detection; UL, unsupervised learning.
Framework of NAD based on UL [45]. NAD, network anomaly detection; UL, unsupervised learning.

Figure 5:

Categorization of UL techniques [23]. UL, unsupervised learning.
Categorization of UL techniques [23]. UL, unsupervised learning.

Figure 6:

Comparison of UL methods on the NSL-KDD dataset. UL, unsupervised learning.
Comparison of UL methods on the NSL-KDD dataset. UL, unsupervised learning.

Figure 7:

Comparison of UL methods on the UNSW-NB15 dataset. UL, unsupervised learning.
Comparison of UL methods on the UNSW-NB15 dataset. UL, unsupervised learning.

Figure 8:

Comparison of UL methods on the CIC-IDS2017 dataset. IDS, intrusion detection systems; UL, unsupervised learning.
Comparison of UL methods on the CIC-IDS2017 dataset. IDS, intrusion detection systems; UL, unsupervised learning.

A relative comparison of this paper with existing surveys/review papers_

ReferencesDiscussionFocus123456
[15]Focus on literature review and performance evaluation of UL methods.IDS≈×××✓×
[16]It provides a comparative analysis of various unsupervised NAD approaches and investigates standard metrics suitable for many connections.AD in networks≈✓××✓×
[17]Presents a detailed review of different clustering methods and their strengths and weaknesses._≈✓××××
[18]Presents a framework for evaluating anomaly detection methods for HTTP.HTTP AD≈≈××✓×
[19]An overview of ML techniques for intrusion detection between 2000 and 2007 is presented.IDS✓××≈××
[20]It provides an overview and comparison of various clustering techniques, pros, and cons.AD≈≈××××
[21]Performance evaluation of four novelty detection algorithms is carried out.Novelty detection✓✓××✓✓
[22]Presents a detailed review of popular traditional and modern clustering approaches._≈≈××××
[1]Nineteen widely employed unsupervised techniques are evaluated to provide an understanding of their performance in various domains.AD≈✓××✓×
[2]The proposed work focuses on a comparative evaluation of four UL algorithms.Smart city wireless networks≈✓××✓×
[3]A survey on ML techniques focusing on unsupervised and hybrid IDS is presented.IDS✓✓×✓×✓
[4]The proposed work aims to provide an experimental comparison of unsupervised methods employed for NAD against five intrusion detection datasets.Anomaly-based IDS≈✓×✓✓×
[23]A comprehensive survey of UL techniques in networking is presented.Anomaly-based in networking≈✓✓××✓
[5]Review and compare ML algorithms for IDS.IDS✓≈××✓×
[6]Presents a detailed review of state-of-the-art IDS methods, datasets, performance measures, and research challenges.IDS✓≈×✓✓✓
[8]An in-depth review of state-of-the-art clustering methods is presented.-≈✓✓××✓
[9]A comprehensive overview of UL techniques for AD is presented, emphasizing their utility in scenarios with scarce labeled data.AD in industrial applications✓✓✓××✓
[11]Review and compare outlier detection techniques from seven different families.Outlier detection≈≈××✓×
[12]AD techniques are explored comprehensively to address the detection of emerging threats.IoT and sensor data✓≈✓××✓
[13]In-depth review of various unsupervised and semisupervised clustering methods.-✓✓≈×✓×
[14]Focus on log file analysis for early incident detection, particularly emphasizing self-learning AD techniques.AD✓≈✓×××
Our surveyThe primary focus is on studying UL NAD methods while considering recent advances.NAD✓✓✓✓✓✓

Relative comparison of UL approaches for network anomaly/intrusion detection_

AuthorsMethodologyAlgorithmDatasetInput dataDoSProbeR2LU2RReal-time detection
[83]Unsupervised approach for outlier detection using subspace clustering and evidence accumulation.DBSCANMETROSECContinuous95958585✓
[84]Tree-based subspace clustering approach for high-dimensional datasets; includes cluster stability analysis.Tree-based subspace clustering (TCLUS)KDDCUP99, TUIDSMixed99968666
[85]Multiclustering scheme incorporating subspace clustering and evidence accumulation to overcome knowledge-based approach limitations.DBSCANKDDCUP99, METROSECN/A----✓
[86]Particle swarm optimization clustering strategy based on map-reduce methodology for parallelization in large-scale networks.PSO clusteringKDDCUP99MixedMax AUC: 96.3
[87]Novel strategy for automatic tuning and optimization of detection model parameters in a real network environment.Clustering and one-class SVMKDDCUP99 and Kyoto UniversityContinuous----✓
[88]K-means clustering to generate training subsets; neuro-fuzzy and radial-basis SVM for classification.K-means, SVM, neuro-fuzzy neural networkKDDCUP99N/A98.897.3197.597.5
[89]Innate-immune strategy via UL for categorizing normal and abnormal profiles.DBSCANKDDCUP99N/AFPR: 0.008, TNR: 0.991, ACC: 77.1, Recall: 0.589✓
[49]Novel approach using cluster centers and NNs to transform the dataset into a one-dimensional feature set.Clustering, KNNKDDCUP99N/A99.6887.613.8557.02
[90]Nature-inspired meta-heuristic approach to optimize the optimum path forest clustering algorithm.Optimum path forest clusteringISCX, KDDCUP99, NSL-KDDN/APurity measure: ISCX: 96.3, KDDCUP99: 71.66, NSL-KDD: 99.8
[91]UL technique for real-time detection of fast and complex zero-day attacks.DBSCANDARPA, ISCXN/AACC: 98.39%, Recall: 100%, Precision: 98.12%, FP: 3.61✓
[92]Modified optimum path forest algorithm to enhance IDS performance, particularly in detecting less frequent attacks.K-means, modified optimum path forestNSL-KDDN/A96.8985.9277.9881.13
[93]Hierarchical agglomerative clustering applied to SOM network to lower computational cost and sensitivity.Hierarchical agglomerative clustering, SOMNSL-KDDMixedDR: 96.66%, ACC: 83.46, Precision: 75, FPR: 0.279, FNR: 0.033

Comparative analysis of network datasets_

DatasetData-sourceYear1234567
DARPA (1998)MIT Lincoln Laboratory1998EmulatedBenchmark✓7,000,000✓DoS, Probe, U2R, R2L41
KDD CUP 99University of California1999EmulatedBenchmark✓4,900,000✓DoS, Probe, U2R, R2L41
DEFCONN/A2000RealBenchmarkN/AN/A✓N/AFlag traces
LBNLLawrence Berkeley National Laboratory2004N/ABenchmark✓>100 hr×Malicious tracesInternet traces
KyotoSong et al. (2006) [114]2006RealBenchmark✓N/A✓Normal and attack sessions24
CAIDAJonker et al. (2017) [118]2008–2017RealBenchmark✓Huge×DDoS20
CDXUnited States Military Academy2009RealReal-lifeN/A5771✓Buffer overflow5
NSL-KDDTavallaee et al. (2009) [126]2009EmulatedBenchmark✓148,517✓DoS, Probe, U2R, R2L41
ISCX 2012Shiravi et al. (2012) [117]2012RealReal-life✓2,450,324✓DoS, DdoS, Bruteforce, InfiltrationIP flows
UNSW-NB15Moustafa and Slay (2015) [119]2015EmulatedBenchmark✓2,540,044✓Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode, Worms49
CIDDS-001Ring et al. (2017) [120]2017EmulatedBenchmark✓31,959,267✓Ping scanning, Port scanning, Brute force, and DoS14
CIDDS-002Ring et al. (2017) [120]2017EmulatedBenchmark✓16,161,183✓Ping scanning, Port scanning, Brute force, and DoS14
CIC-IDS2017Sharafaldin et al. (2018) [121]2017EmulatedBenchmark✓2,830,743✓DdoS, Dos, Botnet, BruteForce, Infiltration, WebAttack, Port scan80
CSE-CICIDS2018Bharati and Tamane (2020) [122]2018EmulatedBenchmark✓16,232,943✓DdoS, Dos Botnet, BruteForce, Infiltration, WebAttack, Port scan80
CICDDoS 2019Sharafaldin et al. (2019) [123]2019EmulatedBenchmark✓32,925✓DdoS_DNS, DdoS_LDAP, DdoS_MSSQL, DdoS_NetBIOS, DdoS_NTP, DdoS_ SNMP, DdoS_SSDP, DdoS_SYN, DdoS_ TFTP, DdoS_UDP, DdoS_UDP-Lag, DdoS_WebDDoS76
BoT-IoTKoroniotis et al. (2019) [124]2019RealReal-life✓73,360,900✓DoS, DdoS, Reconnaissance, Theft29
IoT-23N/A2020RealReal-life✓N/A✓Mirai, Torii, Hide and Seek, Muhstik, Hakai, IRCBot, Hajime, Trojan, Kenjiro, Okiru, Gagfyt21

Recent developments in unsupervised-based network intrusion/anomaly detection_

Authors/yearMethodologyAlgorithm/techniques usedDatasetsAttack classMetrics (%)Limitations/future work
Amoli et al. (2015) [91]Real-time intrusion detection using adaptive thresholds.DBSCANDARPA, ISCXDoS, DdoS, POD, SMURF, Mail-bomb, botnet, port scanning-port sweepPrecision: 98.12, ACC: 98.39, FPR: 3.61Future work: Detecting complex attacks, distinguishing flash crowds and DdoS for better clarity.
Zhang et al. (2016) [42]Utilizing One-class SVM for network intrusion identification.One-class SVMNSL-KDDDoS, Probe, U2R, R2LPrecision: 99.3, Recall: 91.61, F-value: 95.18Low DR for minority-class attacks like U2R and R2L.
Landress (2016) [61]Employing K-means clustering, J48, decision tree, and SOM to reduce false positives.K-means, J48, decision tree, self-organizing mapKDDCUP99DoS, Probe, U2R, R2LACC: 98.92Limitation: Increased computational complexity with SOM; explore efficient hybrid techniques for real-time processing.
He et al. (2017) [94]Exploiting SDN for effective anomaly detection.DBSCANKDDCUP99DoS, Probe, U2R, R2LACC: 94.5Future work: Focus on real-time packet clustering for timely detection.
Ariafar and Kiani (2017) [95]Optimizing clusters using GA, K-means, and decision trees.K-means, decision tree, GANSL-KDDDoS, Probe, U2R, R2LDR: 99.1, FAR: 1.8Evaluate on modern-day datasets like CIC-IDS 201 and CSE-CICIDS2018.
Bigdeli et al. (2018) [96]Incremental cluster updates with spectral and density-based clustering.Spectral and density-based clusteringKDDCUP99, NSL-KDD, Darpa98, IUSTsip, DataSetMeDoS, Probe, U2R, R2LDR: 94%, FAR: 4%Introduce concept drift while merging clusters.
Almi'Ani et al. (2018) [97]Hierarchical agglomerative clustering using k-means for reduced training time.Hierarchical agglomerative clusteringNSL-KDDDoS, Probe, U2R, R2LDR: 96.66, ACC: 83.46, Precision: 75, FPR: 0.279, FNR: 0.033Low accuracy due to low sensitivity toward normal behavior; investigate fuzzy C-means clustering for accuracy enhancement.
Zhou et al. (2019) [98]Hybrid technique using KPCA and ELM.KPCA, ELMKDDCUP99DoS, Probe, U2R, R2LDR: DoS: 98.96, Probe: 98.54, R2L: 94.72, U2R: 36.54, Acc: 98.18, FAR: 2.38The DR of U2R is quite low. To optimize the results of ELM, evolutionary algorithms need to be applied.
Choi et al. (2019) [99]Extracting key features using AE.AENSL-KDDDoS, Probe, U2R, R2LACC: 91.70Improve U2R DR; apply evolutionary algorithms to optimize ELM results.
Aliakbarisani et al. (2019) [74]Formulates a constraint trace ratio optimization problem for Laplacian Eigenmap strategy.Constraint trace optimization, PCA, LDANSL-KDD, Kyoto 2006+DoS, Probe, U2R, R2LACC: 97.84, F-score: 0.878, FPR: 0.001Train and test with different datasets; explore ensemble strategies based on UL models for enhanced performance.
Paulauskas and Baskys (2019) [72]Employing HBOS mechanism for identifying rare attacks like U2R and R2L.HBOSNSL-KDDDoS, Probe, U2R, R2LF-measure: 87Investigate online learning metrics for newly discovered attacks.
Hwang et al. (2020) [100]CNN and AE for extracting raw features from network traffic.CNN, AEUSTC-TFC 2016, Mirai-RGU, Mirai-CCUSYN flood, UDP flood, ACK flood, and HTTP flood, Mirai C&CACC: 99.77, Precision: 99.93, Recall: 99.17, F1 measure: 99.55, FNR: 0.02, FPR: 0.83Improve performance rate of majority classes.
Zavrak and Iskefiyeli (2020) [101]Deploys flow-based IDS with One-class SVM as anomaly detector.AE, VAE, One-Class SVMCIC-IDS2017DoS slow loris, DoS slow HTTP-test, DoS Hulk, DoS golden eye, Heart-bleedAUC: VAE: 75.96, AE: 73.98, One-Class SVM: 66.36Consider flow-based attributes collected at specified time intervals to improve DR and reduce FAR.
Truong-Huu et al. (2020) [79]Uses GAN strategy to extract useful features and proposes a traffic aggregation technique.GANUNSW-NB15, CIC-IDS2017Backdoors, Exploits, Worms, Shellcode, generic, DoS slow loris, DoS slow HTTP-test, DoS Hulk, DoS golden eye, Heart-bleed

  • UNSW-NB15: Precision: 0.84, Recall: 0.85, F1-score: 0.85, AUPRC: 0.8831

  • CIC-IDS2017: Prec: 0.8260, Recall: 0.8268, F1-score: 0.8264, AUROC: 0.9529, AUPRC: 0.8271

Investigate multi-class classification approach for identifying different types of attacks.
Prasad et al. (2020) [56]Proposes a novel cluster center initialization approach to overcome shortcomings of conventional clustering techniques.K-means clustering, cluster center initializationCIC-IDS2017DoS slow loris, DoS slow HTTP-test, DoS Hulk, DoS golden eye, Heart-bleedDR: 88, FR: 88.5, Precision: 88, F-measure: 0.531, ACC: 88.6Address limitations like manual pre-processing and time/ space complexity in MANET deployment.
Megantara and Ahmad (2021) [102]Utilizes feature importance and data-reduction techniques to improve the prediction performance of NAD.Decision tree, LOFNSL-KDD, UNSW-NB15DoS, Probe, U2R, R2L Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode, worms

  • NSL-KDD: Acc: 99.73

  • UNSW-NB15: 91.86

The size of the LOF cluster can be optimized and the threshold value for handling outliers can be improved.
Liao et al. (2021) [103]Presents an ensemble of UL schemes based on a novel weighting scheme.AE, GANsUNSW-NB15, CIC-IDS 2017Backdoors, Exploits, Worms, Shellcode, generic, DoS slow loris, DoS slow HTTP-test, DoS Hulk, DoS golden eye, Heart-bleed

  • UNSW-NB15: Precision: 97.9, Recall: 92.4, F1-score: 94.9

  • CIC-IDS2017: Precision: 83.8, Recall: 84, F1-score: 83.5

Address suboptimal results for specific attack families, explore optimization for better performance.
Verkerken et al. (2022) [71]Proposes an inter-dataset evaluation approach for ensemble UL algorithms.PCA, IF, AE, One-class SVMCIC-IDS 2017, CSE-IDS 2017DoS slow loris, DoS slow HTTP-test, DoS Hulk, DoS golden eye, Heartbleed

  • ACC: PCA: 90.9, IF: 91.1, AE: 99.9, One-class SVM: 98.9

  • AUROC: PCA: 93.73, IF: 95.8, AE: 97.75, One-class SVM: 94.20

Consider employing supervised learning approaches for generalization strength validation.
Singh and Jang-Jaccard (2022) [104]Proposes a unified AE architecture based on CNN and LSTM to examine spatiotemporal correlations in traffic data.MSCNN, LSTM-based AENSL-KDD, UNSW-NB15, CICDDoS2019Normal, Attack (Binary class)ACC: 97.10, Precision: 95.9, Recall: 96.4, F-score: 96.0 (average)Fine-tune hyperparameters for enhanced model performance using optimization techniques.
Wang et al. (2022) [105]Proposes an ensemble of UL algorithms to address challenges in processing overhead and poor detection performance for unseen threats.AE, IFCES-CICIDS2018, MQTT-IOT-IDS2020Sparta SSH brute force, DoS attacks-SlowHTTPTest, DoS-Hulk, DdoS attack-LOIC-UDP, DdoS attacks-LOIC-HTTP, Brute force-XSS, Brute force-web, FTP-brute forceAverage ACC: 96.43, Average Recall: 95.95, Average F-score: 96.02Improve DR of specific attacks like “DoS-Hulk” in the proposed work.
De C. Bertoli et al. (2022) [106]Presents an NIDS for generalized detection performance in heterogeneous networks.Deep AE, FL, Energy Flow ClassifierUNSW-NB15, CSE-CICIDS2018, BoT-IoT, ToN-IoTNormal, Attack (Binary class)BoT-IoT: ACC: 93, Recall: 93, Precision: 99, F-score: 95 ToN-IoT: ACC: 85, Recall: 85, Precision: 87, F-score: 77, UNSWNB15: ACC: 97, Recall: 99, Precision: 57, F-score: 73 CSE-CICIDS2018: ACC: 98, Recall: 88, Precision: 92, F-score: 90Deploy the proposed approach in distributed NIDS for robustness evaluation.
Eren et al. (2023) [107]Proposes a novel tensor decomposition method to enhance the detection of unseen attacks in network anomaly and intrusion detection, leveraging tensor factorization for improved generalization and identification of evolving threats.Unsupervised DL, Tensor factorization algorithmNeris-botnet, Spam e-mail-ROC-AUC, PR-AUC Average ROC-AUC: 0.9661, Average PR-AUC: 0.9152Using a tensor decomposition algorithm with latent factors enables enhanced initialization of the tensor factorization algorithm for improved performance.
Lan et al. (2023) [108]Presents a novel framework for unsupervised intrusion detection that transfers knowledge from known attacks to detect new ones using a hierarchical attention-based triplet network and unsupervised domain adaptation.Attention network, unsupervised domain adaptionISCX-2012, UNSW-NB15, CIC-IDS2017, CTU-13HTTP DoS, Brute Force SSH, PortScan, Brute Force, Dos slow loris, Web attack XSS, Neris, Rbot, Fuzzers, Generic, Shellcode, ExploitsAverage ACC: 96.38, Average DR: 94.16Utilize tensor decomposition algorithm with latent factors for enhanced initialization and performance.
Boppana and Bagade (2023) [109]Synergy of GANs and AE for unsupervised intrusion detection in MQTT networks.AE, GANMQTT-IoT-IDS2020Normal, Attack (Binary class)F1-score: 97Evaluate generalizability to diverse network architectures in future work.

Results of 13 selected UL methods for NAD on the CIC-IDS2017 dataset_

AlgorithmsFamilyAccuracyPrecisionRecallF1-score
K-NNDistance-based96.496.796.097.0
ODINDistance-based97.697.497.598.0
LOFDensity-based9695.393.694.8
COFDensity-based93.393.29593.7
K-meansClustering-based90.287.488.689.5
DBSCANClustering-based93.792.790.692.3
EMClustering-based94.393.291.292.4
PCADimensionality-reduction96.395.494.995.1
KPCADimensionality-reduction97.397.296.997
ICADimensionality-reduction96.396.29696.3
HBOSStatistical-based95.994.993.593.0
One-class SVMClassification-based98.398.19998.4
IFClassification-based99.699.599.299.4

Hardware and software specifications_

ComponentsSpecifications
Operating systemWindows 11
System type64-bit operating system, x 64-based Processor
Processor11th Gen Intel ® Core (TM)
i5-1135G7 @ 2.40 GHz
2.42 GHz
RAM8 GB
Python3.7
DatasetsNSL-KDD, UNSW-NB15, CICIDS2017

Results of 13 selected UL methods for NAD on the NSL-KDD dataset_

AlgorithmsFamilyAccuracyPrecisionRecallF1-score
K-NNNeighbor-based98.498.097.998.5
ODINNeighbor-based99.499.098.698.8
LOFDensity-based97.096.895.395.6
COFDensity-based94.094.596.195.1
K-meansClustering-based9492.4692.393
DBSCANClustering-based94.594.092.994.0
EMClustering-based95.09392.994.0
PCADimensionality-reduction97.897.096.797.0
KPCADimensionality-reduction99.299.098.998.6
ICADimensionality-reduction98.097.597.898.1
HBOSStatistical-based94.493.995.896.0
One-class SVMClassification-based99.499.299.299.5
IFClassification-based99.999.899.699.7

Results of 13 selected UL methods for NAD on the UNSW-NB15 dataset_

AlgorithmsFamilyAccuracyPrecisionRecallF1-score
K-NNDistance-based98.297.897.697.7
ODINDistance-based99.298.998.598.6
LOFDensity-based96.995.895.195.0
COFDensity-based93.894.295.994.7
K-meansClustering-based93.792.092.192.9
DBSCANClustering-based94.293.892.793.5
EMClustering-based95.592.692.393.4
PCADimensionality-reduction97.296.997.397.1
KPCADimensionality-reduction99.198.598.598.7
ICADimensionality-reduction97.797.497.197.3
HBOSStatistical-based92.391.994.694.8
One-class SVMClassification-based99.299.099.299.4
IFClassification-based99.799.599.399.6
DOI: https://doi.org/10.2478/ijssis-2024-0016 | Journal eISSN: 1178-5608
Journal RSS Feed
Language: English
Submitted on: Nov 24, 2023
|
Published on: May 19, 2024
Published by: Professor Subhas Chandra Mukhopadhyay
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year
Keywords:
unsupervised learning,
machine learning,
unsupervised anomaly detection techniques,
network anomaly detection,
intrusion detection
Related subjects:
Engineering,
Introductions and overviews,
Engineering, other

© 2024 Niharika Sharma, Bhavna Arora, Shabana Ziyad, Pradeep Kumar Singh, Yashwant Singh, published by Professor Subhas Chandra Mukhopadhyay
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Previous articleVolume 17 (2024): Issue 1 (January 2024)Next article