Machine learning tools for detecting and visualizing attacks on computer systems

This dissertation focuses on three aspects of computer network security. First, the dissertation introduces several novel visualization approaches that can effectively be used in network anomaly detection research and development. It introduces advanced visualization methods using machine learning, statistical and artificial intelligence sciences. These methods include Principal Component Analysis, Self-Organizing Maps, k-means clustering, hierarchical clustering, Independent Component Analysis, Bi-plots, Mosaic plots and stars plots. The methods are compared in terms of their computational complexity, visual effectiveness and inherent ability to detect selected network attacks. One method that uses Principal Component Analysis for dimensionality reduction of input data and Bi-plots for visualization achieved 100% detection rate given a proposed threshold criterion. Other methods like Self-Organizing Maps and k-means demonstrated the ability to effectively distinguish normal traffic from anomalous traffic using effective graphical means.; Second, it introduces a proposed unified software environment for developing, testing and evaluating network anomaly detection systems. This is aimed at addressing the lack of a homogeneous platform for developing and testing these systems. The proposed environment uses the S Language as a unified platform for developing an anomaly detection system with extensive visualization capabilities. The strength of the proposed system is demonstrated by using several machine learning and statistical methods to detect and visualize selected network attacks. The results of evaluating seven exploratory multivariate analysis methods for anomaly detection and visualization show the effectiveness of the proposed platform in streamlining the processes involved in developing and testing anomaly detection systems.; Finally, low performances issues associated with software-based clustering methods are addressed by developing a hardware-based approach to anomaly detection. A circuit is developed using synthesizable Verilog Hardware Description Language that implements the k-means clustering algorithm in hardware. The circuit clusters network packet information into normal and anomalous traffic and generates an interrupt to indicate that the clustering process has finished. The circuit performance is 300 times faster when measured against a typical software-based version of the algorithm. The circuit was synthesized to produce a total gate count of 50K gates and can run with a clock frequency of 40MHZ.