Research And Implementation Of Middleware Message Based On Netty

In recent years,due to the rapid development of information technology and the mobile Internet industry,the amount of data accumulated by users around the world has grown exponentially,and the era of big data has come.This will not only bring about tremendous changes in the field of scientific research,but will also profoundly affect all aspects of our future life.Currently,the types of computing data in large-scale distributed big data processing system clusters are various.According to their needs and calculation methods,they can be mainly divided into offline processing based on Hadoop and online real-time streaming data processing based on Spark.The online real-time stream data processing is mainly directed to the real-time data collected by the cluster from the front-end monitoring data or crawler system.Unlike the offline data processing,the real-time stream data processing is characterized by dynamic and continuous characteristics.Due to these characteristics of real-time streaming data,large data processing systems must analyze data traffic when analyzing and processing streaming data,and dynamically adjust the data flow characteristics.Therefore,the real-time processing power of data has become a key indicator to evaluate the real-time processing of the system.Message middleware can store and forward massive and complex data to corresponding processing nodes.It is precisely this way that the ability to store and forward messages can cope with the demands of massive data processing.In the face of the needs of message middleware technology in the era of big data,the purpose of this paper is to propose and implement high concurrency and high reliability based on the protocol standards,distributed network communications,and network I/O technologies involved in message middleware.Characteristic and load balanced distributed messaging middleware system.The main work of this dissertation is as follows:1.In order to realize the high concurrency of the message middleware,the message middleware mainly studies the zero copy technology of event-driven Netty high-performance network programming framework,based on the high-performance network programming framework Netty.The network layer is implemented,mainly including the communication protocol.Then,based on Netty,the communication server and the Netty-based communication client are implemented.2.In order to achieve the reliable character of the system message,the Broker persists the message to the harddisk.This article compares database and file persistence solutions,applies a file-based persistence scheme to the message middleware,and proposes a corresponding implementation method;3,corresponding to the system load balancing characteristics,the message middleware passes two To ensure that: Producer achieves a balanced distribution of messages based on the content of the message and the number of Brokers.The rebalancing mechanism implemented by the Consumer allows the dynamic balance between the Consumer and the partition to achieve load balancing and reduce the pressure on the Broker.On the basis of completing the basic message middleware design,this article improves the Nettyd communication model and uses Java AIO based on the asynchronous I/O model to implement the I/O part of Netty.In terms of message persistence,various methods of persistence are studied,and a file persistence method based on the index of the hopping table is proposed.