Research On Distributed Stream-Processing System

The research on streaming data is one of the hot topics among the database circle all over the world recently. During the past three decades, conventional database technologies have been well developed and widely applied. Unfortunately, they could not be adopted to handle a new kind of data, named streaming data, which is generated from applications such as network routing, sensor networking, stock analysis, etc. Because of the rapid data arriving speed and huge size of data set in stream model, novel algorithms that only require seeing the whole data set once are devised to support aggregation queries on demand.Many stream-based applications are naturally distributed. Applications are often embedded in an environment with numerous connected computing devices with heterogeneous capabilities. As data travels from its point of origin downstream to applications, it passes through many computing devices, each of which is a potential target of computation. Furthermore, to cope with time-varying load spikes and changing demand, many servers would be brought to bear on the problem. In both cases, distributed computation is the norm.The architectural issues facing the design of large-scale distributed stream processing systems, and analyses several key techniques in distributed computing are described in this paper. Several disadvantages in existing distributed stream processing systems were pointed out. A new concept which combines component-based middleware and the openness of the CORBA standard is presented to develop a distributed stream processing system in this paper. In order to development facilely, several full-blown technologies were used, in addition to the existing centralized stream processing system. Then the overall framework and the system architecture of the distributed stream processing systems are designed based on ACE-TAO. Simultaneously, the configuration and function is described for each of components in this framework. Finally, this paper discusses novel approaches for addressing load management, high availability. The design which presented in this paper takes full advantage of the existing stream processing and distributed computing technologies. The approaches for addressing load management, high availability are prone to implement. The whole design has the higher application value.