Study On Operation Sharing Optimization Algorithm In DSPS

With the emergence and rapid development of social network, Internet of things, wireless communication network, stream data become an important big data type, which is widely applied in the field of stock trade, electronic commerce, social network and monitor. Compared with the common big data, the stream data has the characteristics of continuous, dynamic and real-time, and new process requirements is proposed:(1) stream is continuously arrived, which need online processing. Job is running for a long time, processing mode is One Pass, the data can`t be stored, but the intermediate results can save;(2) data is changed online, which require topology dynamic adjustment;(3) real-time or near real-time processing.The existing work carry out a lot of research based on the stream processing system and related technologies, form a distributed streaming technology, and realize mass stream high-speed processing. But there are some inadequate about resource allocation problems in DSPS, failed to make full use of the sharing characteristic about operations between multiple jobs, which lead to data redundancy processing and resource waste.Stream job is typical workflow pattern, which is consists of multiple data processing operations. The basic processing operations is similar among different jobs, such as filter, join. So there is a possibility of sharing operation. At present, the operation sharing work is mainly from the view of the job topology structure considering the possibility of share, and establishing operation sharing scheme. The work of sharing operation resources allocation problem is not sufficient, mainly reflected in:(1) the lack of system model, describes the change of resource constraints in operations sharing process;(2) after operation sharing, the output bandwidth resource allocation problem is without sufficient research;(3) the dynamic stream data lead to change of topology, which need to further study on resources allocation quickly adjustment problems.In this paper, we propose an operation sharing optimization algorithm for distributed stream processing. The main work is as follows:(1) Establish operation sharing model for stream jobs. Support the same operation logical sharing between different jobs, and meet the different resource constraints, including computing resources and physical network bandwidth resources.(2) Study resource allocation problem under multi resource constraints. Based on the similarity between operations, a sharing decision algorithm is proposed to find the logical sharing operations. The resource allocation process is abstracted into a bin packing problem, considering the operation resource usage after share, the computing resources is combined and the output bandwidth is accumulated of sharing operation, which is compared with share nothing.(3) Study the resource mapping fast adjustment problem. New job dynamic addition leads to the change of overall topology and the sharing operation resource demand, screen the sharing operation according to the current configuration of resources, and select the operation with maximum benefit, using less adjustment of consumption,, quickly meet new resource demand, ensure the real-time and dynamic of stream processing.(4) Establish analogous experimental platform. Verifying that operation sharing can effectively save computing resources, and meeting the different system resources constraints on sharing operation number. Verifying the effectiveness of operation sharing algorithm, quick adjustment algorithm is better than the resources re-allocation algorithm.