Research On Optimization Of High Concurrency Message Mechanism Based On Kafka

The continuous development of informatization and the emerge cloud computing and distributed technologies recently brings a new era of big data.In the meteorological area,the advancement and expansion of business services,the continuous encryption of observation frequency,the enrichment of processed products,and the explosive growth of meteorological data requires new technology to manage and process the massive meteorological data.Among these technologies,Hadoop,Map Reduce and other technologies has attracted increasing interest,but high latency and slow response is hindering the development.In order to meet the needs for real-time and high-speed processing of massive data,various message-oriented middleware such as Active MQ and Kafka have been intensively studied,which offer solutions to resolve the problem of massive real-time data access and processing.However,how to optimize the performance of various message-oriented middleware and further improve its concurrency performance and throughput is still the research hotspot.Aiming to solve the problem of insufficient concurrent processing performance of massive real-time meteorological data,a comprehensive investigation and analysis of recent research status in both China and overseas.Through a detail comparison of different message-oriented middleware,Kafka,a distributed message subscription system,was selected as the research object,and a distributed architecture is constructed.Based on the entropy weight method,we carried out research on the classification strategy of weather monitoring data,and conducted experimental comparison and analysis to optimize the strategy configuration of Kafka.The specific research content in this paper are listed as follow:(1)Detailly compared the characteristics of various message middleware,such as Active MQ,Rabbit MQ,Kafka,we build an experimental model based on Kafka,has the ability to process massive amounts of weather monitoring data in real time.(2)In order to find out the resource requirements of different kinds of meteorological data in concurrent processing,we proposed an algorithm to analyze the key characteristics of different types of meteorological data using the entropy weight method,carried out the scientific classification strategy of meteorological data,as the basis of creating Kafka partitions,Improve the concurrency performance of Kafka.(3)The key features,such as Thread,Partition and Replica,have been selected to investigate the impact on Kafka concurrency performance and throughput with the aim to optimize the configuration strategy that is able to adapted to massive meteorological data processing with the objective to further improve concurrency performance and throughput of Kafka.(4)The monitoring information related with the actual application scenario is designed as an experimental data set,and the optimized experimental model is analyzed and verified using the test tools on the Producer and Consumer.The experimental results demonstrate that the strategies proposed in this paper can achieve the optimal performance under the condition of limited basic resources to satisfy the demand for high concurrent processing of massive meteorological information monitoring.