Research On Service Chaining Method Based On Locality And Instant Correlation Of Big Data Streams

In recent years,there has been great progress made in research and practice in data stream processing systems.The classic systems already have the ability to process massive data streams in near real time.However,there are still some shortcomings:(1)most systems usually collect original large-scale data streams with low value density,which will lead to degradation in storing,processing and searching capacity;(2)most systems rely on prior coding to develop global data stream processing logic,which however is in contradiction to the nature of locality and uncertainty in data streams;(3)the traditional ‘request/response' mode limits the system's automation and responsiveness.Targeting the nature of uncertainty in data streams,this thesis undertakes further research based on proactive data service model.The main contributions are as follows:1.This thesis proposes a service hyperlink model adapted to the proactive data service model,and presents a decentralized manner on processing data streams by avoiding presetting global logic.The experimental results show that the hyperlink number is controllable as the dataset size increases and the parameters change.The thesis proposes a service hyperlink generation algorithm based on historical data.It transforms the service hyperlink generation into a time-constrained frequent co-occurrence pattern mining problem.A GFE(Generate-Filter-Expand)strategy is proposed to effectively generate service hyperlinks.Experimental results prove that the spliced logic based on service hyperlinks can produce results that meet thermal power plant fault detection application requirements.2.The thesis proposes a data-driven service hyperlink updating algorithm,which can iteratively learn and update service hyperlinks at runtime.The experimental results show that,the hyperlink number is controllable with the continuous arriving of the data streams.To enhance the efficiency,it proposes a landmark-based space-freeing strategy by expanding the classic Lossy Counting algorithm.It also optimizes a data structure to quickly locate the storage location of a service hyperlink that needs to be modified or added.The experimental results prove that the optimization can at most reduce 65% of average execution time under different arrival rates,62% under different parameters.3.The thesis proposes a service-hyperlink-based service chaining method to solve the problem in data stream uncertainty.It proposes a hyperlink selection strategy based on locality and self-adaptation.The strategy can reduce the selected hyperlink number,avoid repeatedly propagating events,and improve chaining efficiency while guaranteeing correctness.The experimental results show that the service hyperlink is an effective auxiliary means:(1)it can correctly assist proactive data services in chaining service and cover 91.04% of the thermal power plant fault detection application requirements;(2)it can also improve the chaining efficiency.It can at most reduce 51% of the average latency under different service numbers,and 48% under different arrival rates.Targeting the current challenge in the data stream uncertainty,based on generation and updating of service hyperlinks,this thesis uses service hyperlink to flexibly select and splice local processing logic at runtime,resulting in producing results that meet application requirements from a locality point of view.The validity of the method is verified by theoretical proof and experimental analysis.