Design And Implementation Of Internet Of Things Data Acquisition Platform Based On Netty And Kafka

In recent years,with the vigorous development of Internet technology and intelligent sensor industry,the Internet of things technology that integrates the two is becoming more and more mature.More and more traditional industries are carrying out business innovation and industrial upgrading with the help of this technology,and the number of Internet of things devices is also increasing sharply.When massive devices are connected to the network,how to connect massive IOT devices to the data platform and collect data in real time and efficiently is a difficult problem.Therefore,this paper designs a data acquisition platform of Internet of things based on Netty and Kafka.In terms of device connection,Netty communication framework is based on event driven network,with high concurrency and strong customization ability.It can flexibly expand the system.In terms of data processing,Kafka has the ability of message subscription and persistence,and has greater throughput than traditional message oriented middleware.On this basis,aiming at the realtime problem brought by Kafka message middleware,this paper proposes a data processing algorithm based on multi-level priority queue to process data together with Kafka.The algorithm takes the product of data priority and length as the standard.First,the data is assigned to the multi-level priority queue and short job priority queue,and then the queues are created in turn according to the data processing in the multi-level priority queue until all the data are processed.Aiming at the problem of sticking and unpacking TCP protocol,a user-defined communication protocol is introduced to improve the data transmission speed of the platform.To solve the problem of device connection waiting caused by the full IO thread pool of Netty,this paper proposes a thread pool capacity configuration scheme,which considers the waiting time of device connection and the processing time of blocking IO,and dynamically adjusts the IO thread pool of Netty within a reasonable range.Finally,based on the optimized platform,starting from the device end and the service end,this paper builds a fully functional Internet of things data acquisition platform,and tests the functions of the data acquisition platform.The results show that the test results of each function can meet the expectations,and the platform can also provide various services while collecting data.