| With the advent of the era of massive data,the amount of data required to be processed by computers endorse a rapid increase.Faced with the ever-increasing data size and computing requirements in the area of Internet and industrial Internet of Things,a single node can no longer handle the requirements of data storage or data processing properly.In such a context,Apache Spark,a distributed computing framework with good environmental adaptability and computing performance,is widely used to address such challenge.Spark itself has dozens of parameters which can affect the performance of the system.By adjusting these parameters,the parallelism of task execution and the memory used when processing data can be affected.Thus these parameters have a significant impact on the performance of Spark applications.typically,these parameters are adjusted manually.However,due to the amount of tunable parameters and the differences between the application and the physical cluster environments,there is a clear restriction on manual tuning methods which depends on the professionality of administrators and requires repeatedly efforts on different hardware environments.Thus,the manual tuning method has certain limitations to meet the users’ need in all scenarios.To solve the above problems,this paper does research on how to automatically adjust the parameters of Spark applications.Our work separately studies how to adjust the parameters base on the information of application runtime logs,and how to automatically optimize the parameters in the absence of historical data.Further more,this paper present a statistical method to further refine the optimization process and made discussion on its applicability.Finally,we present a system based on these algorithms proposed above which works to find a better parameter for the Apache Spark deploied with on yarn mode.If there are sufficient history application records,our system will try to model the relationship between the system runtime parameters and the system performance with classic regression methods.The prediction results of this model are used to tell whether a parameter is good or not.Finally,a suitable parameter is found under the guidance of the model in a user specified space to optimize the system performance.In the scenarios where there are not enough history information to train a model,this paper turns to the heuristic method to adjust the application parameters leveraging the hardware configuration of the system,so that our application can obtain a configuration with improved performance compared to the default ones.A strategy to smoothly combine these two methods is designed in order to allow our system to make use of both of them.Finally,we study the impact of the accuracy and speed of the application performance model on the final optimization effect.According to experimental and theoretical results,we present a strategy on choosing the proper application performance model to further refine the optimization result.The experimental results show that the performance of Spark applications is significantly improved with and without historical data.Our system reduced the run time of four benchmark applications by about 59.92% on average when there are sufficient historical data. |
