Research On Pivotal Tenchnologies Of Massive Concurrent Data Processing Based On Cloud Computing

The rapid development of cloud computing technologies led to the increasing internet users.The number of transactions on the Internet is increasing correspondingly,and the network data is showing an exponential growth.The massive data that appears on the Internet needs to be processed urgently.There will be problems such as poor concurrency and low efficiency are the problems causing by using traditional data processing techniques for massive data processing.On the one hand,this paper explores load balancing,improves the weighted Round-Robin algorithm,and calculates the weights dynamically according to the operating status of the server.After the improvements,the weighted Round-Robin algorithm has improved in response time,throughput,and actual concurrent amount.On the other hand,this paper proposes a solution for mass small-file data storage.It adds an independent small-file data storage module in HDFS.The efficiency of HDFS for storing mass small-file data is improved,through small-file merging,creating index and data cache,and the extensibility of HDFS is not affected.This paper verifies that the improved weighted Round-Robin algorithm can improve the performance of the algorithm at a certain extent through the load test tool LoadRunner simulation experiments based on the theoretical research.Through the experimental test for the memory consumption of metadata nodes and the efficiency of reading and writing small files,the improved HDFS is more effective.It has a certain guiding significance for the research on the storage of small files for the research in the following days.