Design And Implementation Of Data Acquisition System Based On Streaming Processing

In recent years,the rapid development of mobile Internet and Internet of things pushed global explosive growth of data.Due to low cost sensors and mobile devices,the amount of sensory data generated in all aspects of production and life in human society are growing.This kind of data is not only large in size and scale,but also has the characteristics of real-time arrival,continuous uninterrupted and fast frequence.It is often classified into "streaming data".The basic services of streaming data are faced with increasingly severe challenges.However,the existing data acquisition tools are cumbersome,developers have high cost of learning and management,lack of real-time assurance.We design and implement a data acquisition system based on parallel stream processing in distributed environment.Raft algorithm is a distributed consensus solution used in streaming data acquisition system,and we rewrite the function of Akka Cluster's Distributed Subscribe Publish,the ACK confirmation mechanism is added to the process of message broadcast.This system supports distributed and multiple level deployment,as an important part of the streaming service infrastructure.Starting from the function and performance requirements of the data acquisition system,formulate the overall design scheme of the system.According to the function module of the system and follow the design concept of "micro service",this system is divided into security authentication module,subscription management module,data access module,system state module,data storage module and data distribution module.Each module is independent,and in actual deployment,it realizes the separation of physical logic level,decoupling each module,improves the flexibility,reusability and disaster tolerance of the system.Finally,each function of this system is tested based on processing and verification system to meet the expected goals.Then we summarize the streaming data acquisition system,put forward some optimization problems which need to be solved or improved,and make the prospect of the future work.