Research On Load Balancing Of Netty Internet Of Things Server Cluster Based On Kafka

In many IoT server applications,improving the concurrent performance of server systems to support mass terminal equipment is one of its important tasks.In order to meet the requirements of low latency and high throughput when the system processes large volumes of data,this article optimizes the system from both vertical and horizontal aspects.Vertical optimization is to improve server processing capacity by rationally allocating system resources from a stand-alone perspective.However,in the current Internet environment full of massive data,a single machine cannot meet the demand at all,so it is necessary to build a distributed system through multiple machines to improve the processing capacity of the service.In order to further improve the performance of the distributed system,it is necessary to adopt a horizontal optimization method to improve the availability of the cluster from the perspective of load balancing and improve the balance rate of resource utilization.This article optimizes the traditional IoT server system based on the above two optimization directions.The main research work is as follows:(1)This article first details the internal principles of the network I/O model,including the thread models of Reactor and Netty.Then it introduces the design principle of Kafka message middleware,and discusses and analyzes it from the two aspects of the architecture and message queue.Then,the detailed steps of Kafka cluster and its load balancing algorithm are introduced.(2)Aiming at the key problem of coupling between Netty's NIO thread and business processing thread,a design scheme of Kafka-based IoT terminal service system was proposed.Optimize from three aspects:data processing module,message middleware module and business processing module.This solution combines Kafka message middleware to decouple Netty's NIO threads from business processing threads to solve the problem of terminal request accumulation caused by Netty NIO threads being blocked by business logic.This article also proposes a thread pool tuning technology.By optimizing the calculation method of the thread pool capacity,the optimal number of threads in the system is obtained.This thread pool technology is applied to the consumer side of Kafka,which greatly speeds up the consumer.Message processing speed to prevent message middleware from accumulating messages.Compared with the traditional Netty and kafka server system solutions,the simulation results of the proposed solution have great advantages in terms of average response time and throughput in high concurrency situations.(3)Aiming at the shortcomings of the load balancing algorithm of the native Kafka consumer cluster,an adaptive load balancing algorithm is proposed.This algorithm calculates the total resource utilization of the system by monitoring the utilization of CPU,memory,and bandwidth in real time.According to the resource utilization Rate for load balancing.The algorithm in this paper is implemented and simulation experiments are performed.The results show that the algorithm in this paper saves system resources more than the native algorithm,and is suitable for consumer clusters with large differences in performance configuration.The research results of this paper have improved the overall data processing capability of the Netty IoT server system,and enhanced the reliability and robustness of the system data processing process.The system designed according to this scheme has been put into use in the smart campus project,and there is no unstable data interaction in the application of actual scenarios,and the system runs well.It can provide a reference for the development of other IoT server systems.