Data Stream Processing In Financial Database

Nowadays, lots of data is modeled best not as persistent relations but rather as transient data streams, like stock market transactions. These data streams have new features: rapid, time-varying, possibly unpredictable and unbounded in size. It is not feasible for traditional database management system to handle these data streams. Applying data stream processing technology to maintain these data incrementally is more convenient. It will avoid the bottle neck problem when doing the bulk updates.Frequent items mining is a very basic but important task in the data stream processing. However the traditional algorithms such as Lossy Counting can only find out frequent items based on computing their counts. In some situations, people want to monitor those items whose weight exceeding a user-specified threshold over the data stream. We propose a novel algorithm to address this problem. The Lossy Weight Algorithm can output an approximate result whose error is guaranteed not to exceed a user-specified parameter. Experimental results show that the new algorithm yields very good performance on both space and time cost.Incremental database maintains data incrementally, which is similar as data streams processing, with more loose time scale and size of updates. It saves the intermediate results of compute-intensive tasks. Thus, it can balance the peak load when to idle time. Incremental database can greatly improve system efficiency and is short in response time. It has great practical significance in financial data processing.This dissertation introduces the new concepts called offline data stream and incremental database. Online/offline data streams and incremental database have different restrictions. All three were analyzed and corresponding optimization ideas were proposed. The novel algorithm can be effectively applied to online/offline data streams and can be used as pre-processing method of incremental database. As a brand new optimization technique, incremental database can be widely used in variety of occasions with data stream features.