Design And Implementation Of A Real-time Data Acquisition And Processing Platform

With the arrival of 5G era,the concept of "everything can be connected" has gradually integrated into our lives,In addition to the traditional industry,IOT that for civil use is becoming more and more mature,more and more IOT platforms gradually pour into the Internet market.But the scale of related projects is massive,and the corresponding IOT equipment is various,How to analyze diverse protocols,how to collect and efficiently process high concurrency and strong real-time data are still the pain points of the existing IOT products.Consider about the perspective of performance and scalability,how to ensure the timely and accurate processing of massive real-time data and simplify the development of horizontal field has become a hot issue.This paper is based on the actual project of the enterprise,designs and implements a real-time data acquisition platform based on asynchronous-event-driven open source network framework and flow processing framework.The platform realizes the collection,processing and storage of real-time data through abstract modeling,unified data structure of equipment,high-performance network framework and stream processing framework,at the same time,the idea of domain model driven design is used to layer the complex system and reduce the coupling degree.In addition,the system solves two difficulties,the first is the real-time data acquisition of mass terminal equipment,which realizes the characteristics of high throughput and low delay,The second is to process real-time data accurately and quickly,and to achieve flexible and scalable data processing for different business needs.In view of the two difficulties mentioned above,this paper mainly does the following work:(1)The high performance asynchronous communication framework netty is used to improve the concurrency on the acquisition side.The data collected by different data are packaged uniformly by configuring the parameters of netty.Then,the data is parsed by the data parsing handler and then pushed to the distributed message queue Kafka.Through the above means to achieve real-time data flow peak clipping,to ensure the real-time and reliability of the data acquisition module.(2)For real-time data processing,the system uses open source stream processing framework Flink.By optimizing the Flink parameters,the delay of data processing is reduced,and the user-defined processing module is extended by the user-defined class loading mechanism,which achieves the dynamic reorganization of the stream processing module,measures above greatly improve the flexibility and scalability of the system.(3)According to the existing Internet of things equipment and basic requirements,complete the demand analysis of data acquisition platform,complete the modeling of demand analysis and related documents,and determine the general structure of the project.At the same time,complete the design of data acquisition and processing platform,including front-end,background and database design,and build the corresponding design model and related documents,according to the above documents to achieve the corresponding project module and coding work.(4)The platform is deployed to test the system function in the actual environment,and the performance test is carried out to verify the functional requirements and non functional requirements of the data acquisition platform.