Research On Anomaly Detection Based On Concept Drift

In the field of Machine Learning, there are always some data which are different from the normal data or against the existing general laws, we called these objects “outliers”, but in practice, these data are often overlooked. The truth is that these data may have special meaning in the field of anomaly detection, since they contain more useful knowledge than the normal data, they are the focus of the research, we can make accurate judgments and solutions by find the laws of these anomalies.In most of systems, the received sensor data is fast, real-time and infinite. The sensor data is transmitted to the data processing center in the form of data stream, which bring a challenge to the traditional anomaly detection techniques. Meanwhile, in data stream, the patterns of behavior may change over time, i.e., normal mode may change, that is, concept drift. How to adapt to concept drift is also the focus of research in the field of anomaly detection.In this thesis, we will build some models to fit the entire sensor data based on data mining and statistical technologies, then use these models to identify anomaly data, which including some single data points or abnormal pattern with a continuous period of outliers.For outlier detection, we use recurrent neural network technology of data mining to do some research, proposes an algorithm of using the RNN(Recurrent Neural Network) to build a model, which can effectively fit the sensor data that is collected irregularly. Due to the timing property of the RNN model, the model has a better fit quality than the polynomial fitting technology and the BP(Back Propagation) neural network fitting model. It can achieve better performance for removing the outliers or noises from huge amount of data.As for the detection of abnormal patterns, we need to take into account the situation of concept drift. We present a new way and method to solve this problem. In this thesis,Markov process will be applied to the detection of abnormal patterns. We proposed an algorithm based on k-means algorithm and Markov process, this algorithm can adapt the situation of concept drift accurately. Then we try to apply this method to simulated data and this project which related to this thesis, and we implement an existing algorithm based on SAX(Symbol Aggregate Approximation), we did a lot of comparative experiments, showed that our method which proposed in this thesis can get more accurate results when detect abnormal patterns which contain concept drift in the data stream.