Design And Implementation Of Anomaly Detection System Based On Log Feature

With the rapid development and widely used of cloud computing technology, application systems are gradually migrated to the distributed computing environment which treats large server cluster as core. Meanwhile,they becomes more and more large-scale and complexity. Detecting system’s abnormal behavior of the system will face great challenge because more information needs to be handled by maintenance staffs when systems are failed. Application system log keeps the execution trace and exists in all components, we can mine log’s information to detect system’s abnormal beheavior.Traditional technology of log analysis and anomaly detection are mature in their respective fields. At the meanwhile they have some disadvantages such as incomplete description problem. Now we need to describe the problem of new ideas.Based on above background, we design and develop an abnormal behavior detection of large-scale system which is based on log feature and uses a combation of system’s source code analysis, process modeling, data mining, machine learning algorithm and outlier detection method. The system, which is based on cloud environment and focus on log analysis, improves the existing technical diffic ulties to adapt to a variety of log formats and stydy the system byhavior in time and logical sequence. It improves a lot of problems such like low accuracy of heuristic methods and low efficiency of frequent pattern mining. Follows are the main implementation of this system. Firstly, by using IDE plugin of Eclipse to analyse system’s source code we can make an index and get abstract syntax tree to generate the log model and process structure which include log format and contex logical relationship. Secondly, we do the pretreatment to the log using by distributed storage environments and computing framework. For the log files, we use log models and process models to transform them by improved pattern matching functions. The result is to be counted and selected to generate process state variable and message count variable to group all the logs and make the eigenmatrix. Then we find PCA which can reduce the matrix dimension and show multivariate data directly to detection of outliers so that it can locate the abnormal behavior of the system. Lastly, we verify the correctness and effectiveness of the proposed method o n Hadoop and C loud Stack systems. Lastly, wo choose decision trees to declare the behavior of system. Decision trees regulate the behavior so that it can provides the basis for determining the legality of the system behavior. While we use a variety of reports for the behavior of the system to get the system operational profile. In the experimental section using Hadoop open source systems to ver ify the correctness and validity of the method by relevant functional testing and performance testing.