Research And Implementation Of Multi-stream Collaborative Processing Technology

At present,the mainstream stream processing framework adopts the design mode of distributed sub-task independent computing.Distributed sub-tasks do not interact with each other during processing,and cannot acquire information elements of other sub-tasks.It is difficult to perform distributed sub-tasks.Collaborative processing.At the same time,there are data with sequential characteristics in some stream processing scenarios,and such data must complete the timing synchronization operation before performing subsequent analysis processing.On the one hand,in order to solve the problem that the stream processing framework is difficult to perform collaborative computing in distributed subtasks,this paper introduces the collaborative processing model based on Apache Storm's stream processing framework,and builds the ability of collaborative computing among distributed subtasks.Firstly,real-time message passing between distributed sub-tasks is realized based on Storm,which is used to provide technical support for collaborative computing operator distribution and data synchronization.Then,the modeling work of distributed sub-task collaborative computing is carried out,and each model is used.The module is designed and implemented to support users to achieve collaborative calculation across subtasks by specifying operator types and configuring operator content.On the other hand,for the timing synchronization problem of multi-stream,this paper designs a timing synchronization module of master-slave structure.The module is implemented by Akka concurrent processing framework,and realizes the message interaction of various roles in the module based on the Actor model of Akka framework,and designs the message type and data structure of the interaction in the module,and determines the master node and work in the module.The process by which each node processes the corresponding message.Based on the above research work,this paper designs and implements a prototype system for real-time processing of multiple data streams.The corresponding experiments prove that the prototype system can realize the synchronous operation of multiple data streams and has efficient collaborative processing capabilities.