Research On Performance Variability Analysis And Simulation Of Cloud Network For Big Data Application

In the early stage of cloud network research,researchers believed that CPU was the main limitation of big data application performance,but through the research,it was found that the variability of big data application performance was basically caused by the low efficiency of the framework,and this problem was solved in later versions.At the same time,due to the multi-tenant characteristics of cloud computing,the operation of other tenants will affect the stability of big data application performance,making their cost difficult to estimate.Therefore,the variability of cloud network performance has become an important factor affecting the performance of modern big data workload.To solve above problems,thesis proposes a big data workload performance optimization algorithm based on VOA(variability optimization algorithm)to reduce the impact of cloud network variability and improve the performance stability of big data applications.The main research contents of thesis are as follows:Firstly,aiming at the variability of cloud network performance,a cloud network performance variability analysis method based on the internal network mechanism of cloud platform is proposed.By collecting the cloud network data from a large-scale commercial cloud EC2 and a small-scale private research cloud vultr,analyze its change and variability level.The analysis indicators selected here are from the internal network mechanism of the cloud platform,including network management mechanism,network access mechanism and virtual network mechanism,and finally get the variability factors affecting the performance of the cloud network.Through the analysis,it is found that for EC2 cloud server,in order to reduce the variability of cloud network performance,we must first choose the server with a high number of initial tokens under the condition of ensuring enough budget.At the same time,network intensive services will instantly run out of tokens,so we should choose simple query services.For vultr cloud server,since there is no limiting mechanism,network intensive services can be selected.At the same time,in order to reduce network jitter and delay as much as possible and improve network bandwidth,VLAN virtual network should be selected to complete data transmission.Then,aiming at the variability of big data application performance,a big data application performance variability analysis method based on token bucket simulator is proposed.In order to eliminate the influence of CPU,bandwidth,memory and I/O,and ensure the accuracy of experimental results,a token bucket simulator is designed to replace EC2.Hibench and Bigdatabench are run on vultr platform and token bucket simulator platform respectively to show the impact of cloud network performance variability on big data applications.Through the analysis,it is found that network management mechanism,network access mechanism and virtual network mechanism are the key factors affecting the performance of big data application.In addition,the sorting time complexity and the time-consuming of shuffle stage in big data computing framework are also the key factors affecting the performance of big data application.Then,according to the results of the first two chapters,the design of big data workload performance optimization algorithm based on VOA is proposed.Aiming at the performance bottleneck of Map Reduce,an algorithm is designed to dynamically select the sorting mode according to the difference of data volume between different files,the number of different key values and the number of different files.This part of the algorithm is combined with the cloud network performance optimization algorithm to obtain the big data workload performance optimization algorithm based on VOA.Finally,a comparative test is designed to compare the performance of the original big data workload with that of the big data workload combined with VOA.The experimental results show that the performance of the big data workload is more stable after the coarsegrained optimization of the network layer and the fine-grained optimization of the application layer.Therefore,the VOA algorithm can effectively reduce the impact of the variability of the performance of the cloud network on the big data workload.