Analysis and Evaluation of Dimensionality Reduction Techniques in the Context of Network Intrusion Detection System

Abstract

The rapid expansion of computer networks has increased the frequency and sophistication of cyberattacks, making Intrusion Detection Systems (IDS) a critical line of defense for protecting network infrastructures. though Machine Learning (ML) has significantly improved the detection capabilities of IDS, modern network traffic often produces massive and high-dimensional datasets. This high dimensionality, commonly referred to as the curse of dimensionality, can lead to high computational complexity, increased model complexity, and overfitting, which may reduce the ability of ML models to accurately detect cyber threats. This study addresses this issue by comparing several feature selection and feature reduction techniques in the context of ML-based IDS. The adopted techniques include ANOVA and Mutual Information as filter-based methods, Random Forest and Lasso L1 regularization as embedded methods, in addition to two hybrid techniques designed to leverage the advantages of simpler feature selection approaches, and PCA. The experimental evaluation is conducted using two well-known IDS datasets, namely CIC-IDS2017 and CSE-CIC-IDS2018. All experiments are based on the XGBoost algorithm, which is widely recognized for its strong predictive performance and effectiveness in classification tasks. The obtained results for both binary and multiclass classification show promising rformance and provide valuable insights into the role of dimensionality reduction in enhancing the effectiveness,efficiency, and generalization ability of ML-based IDS