Research And Implementation Of Data Acquisition And Anomaly Analysis In Dependable Monitoring System

With the continuous development of network technology and the sustainablepromotion of the network applications, the people’s production and daily life areincreasingly dependent on computer networks. However, the network is very fragile andis often subjected to someone’s intentional or unintentional destruction, which results inincurring network security issues. Providing network service with high reliability, highavailability and low cost has become a hot topic. In order to meet these requirements, itdoes not work if it only relies on the software which finds network anomalies after itoccurs. Therefore, it is necessary to develop a dependable monitoring system to predictthe occurrence of network anomalies and provide accurate information for the networksecurity measures. Collecting data on each node is the basis for dependable monitoringsoftware, while anomaly analysis of the collected data is its core function.In order to accomplish the functional modules of dependable monitoring systemwhich are data collection and anomaly analysis, our contributions are as follows:①Reading the relevant files in the/proc file system and obtaining detailed hostmonitoring data by executing the system monitor command of Linux via pipe.②Analyzing the basic principles of the Linux network architecture and a varietyof network packet capture technologies. Because of the low network packet capturespeed, we propose a tagged status buffer algorithm based on the “producer-consumer”model to solve the problem that the speed of data flow from the NIC to the kernel underthe adoption of NAPI is low. And quantitative analysis of packet loss condition ispresented. The capture performance of network packet is greatly improved under thesituation that the packet loss condition is not met.③Designing two data anomaly analysis methods which are static and dynamic,respectively. Static threshold anomaly analysis is implemented by the combination ofmonitoring methods. We adopt the combination of principal component analysis, thejoint Gaussian distribution and the Bayesian optimal classifier technology to implementthe dynamic anomaly analysis. In order to meet the requirements of the large amount ofthe collected data and abnormal analysis with high-sensitivity, the static and dynamicanomaly analysis methods are combined by adopting a specific algorithm.④The data collection module and anomaly analysis module of dependablemonitoring system are tested, as well as the detailed test process and comparative results are showed. According to the test results, the improvements of data collectionmodule and anomaly analysis module are analyzed.In a word, this thesis analyzes and researches the key problems that dependablemonitoring technology confronts in large-scale distributed application environment, andthen designs and modifies the method of improving the network packet captureefficiency, as well as the anomaly analysis technology based on probability statistics.Finally, we analyze and evaluate the effectiveness of these algorithms theoretically, andthe effectiveness is proved by our experimental results.