Intelligent Anomaly Detection in Enterprise Networks Using Machine Learning and Statistical Methods

Main Article Content

Shubh Prabhat

Abstract

Enterprise networks see varied network traffic․ Next-generation signature detection does not detect many network attacks․ Anomaly detection approaches can help by building models of normal network traffic to detect abnormal traffic․ Statistical anomaly detection approaches have high false-negative rates․ Machine learning approaches provide limited insight into why a given network flow is suspicious․ We present a hybrid statistical and machine learning approach to network anomaly detection․ We build four models of normal network traffic using only normal network traffic data‚ a median and median absolute deviation z-score‚ a Mahalanobis distance with a minimum covariance determinant estimator‚ a squared principal component residual‚ and an isolation forest anomaly score․ The four anomaly scores and 30 traffic flow features chosen according to mutual information are used as inputs․ A random forest classifier is used․ Out-of-fold cross-validation is employed to ensure that the test dataset is not used to test models built using the training dataset․ Experimental results on the official training and test datasets of the UNSW-NB15 dataset are compared with the results of other machine learning and statistical methods․ On the test set‚ using 34 features instead of 68 results in an accuracy of 86․87%‚ a detection rate of 98․89%‚ an F1-score of 0․892‚ and an area under the ROC curve of 0․979․ This approach is comparable in performance to other complete random forest models and outperforms statistical models both individually and combined (AUC = 0․778-0․826)․ Evaluation suggests that the statistical model predictions are strong predictors in the combined model but do not add predictive power on top of other flow features․ Evaluation of different prediction thresholds suggests that setting a higher threshold of 0․8 for predictions would lower the false-positive rate from 27․86% to 7․04% at a true-positive rate of 91․70%․ Other findings about differences in cross-validation and cross-partition performance may be useful for business applications․

Article Details

Section
Articles