Change Detection Algorithm For Data Flow In Real-time Exchange Rate Data Flow Anomaly Detection

In the Foreign Exchange Investment Business of commercial banks, exchange rate is a kind of online, losable, limitless data stream. From time to time there are some abnormal data blocks in this stream for some reasons. They will cause economic loss of banks or clients. How to detect these abnormal data blocks in exchange rate data stream is the main problem in the development of foreign exchange investment system. Through the observation and analysis of exchange rate data stream, it is not difficult to find that the most important feature of abnormal data block is that it has different data distribution. The point is that we can discover those errors by detecting the distribution change of data stream. The intention of this thesis is to provide an accurate, efficient algorithm which can detect the distribution change in data stream so that we can use it to detect and filter the abnormal data blocks in exchange rate data stream.Even though there are a lot of developments in the field of data stream processing, many of those focus on how to get the approximate results in data query, mining and analysis of data stream in an environment with limited storage. Very few are concerned with the detection of change in data stream. In the daily practice, however, we often find that human can detect the change in data stream easily. Although human is not able to process huge quantity of data, we can analyze and simulate the procedure to solve our problem. Therefore, this thesis focuses on simulating the human detecting procedure, talks about the theoretical background and provides an algorithm based on it.Firstly, through the observation of the procedure in which human can detect the change in data stream, this thesis discusses it's theoretical background, and then provide a two static sample based change detection algorithm and it's optimization. After that, in order to utilize the algorithm in the environment of data stream, the two slide window based data stream change detection algorithm is provided. In the session of experiment, the result of the algorithm is provided by simulating a data stream with many changes in it. Through these experimental data, this thesis not only proves the veracity and efficiency, but also discusses how to choose related parameters to get the best result in practical environment.In the end of this thesis, the data independent detection model of exchange ratedata stream is built. Then, the algorithm provided in this thesis is used on the model to solve the problem of detecting the abnormal data block in exchange rate data stream. Experimental data are also provided.