Research On Elasticity Resource Management Strategy For Streaming Computing

With the rapid development of computer technology,the requirements of data processing speed in various fields are getting higher and higher and the traditional batch processing technology unable to meet the real-time requirements of data processing.So,the emerging streaming computing technology has drawn more and more attention and real-time computing platform such as Storm has also developed very quickly.However,in a big data streaming computing environment,all applications will run for a long time after they start executing.The resource scheduling in the online environment not only takes into account the current node allocation status,but also considers the degree and trend of data flow changes and the impact of scheduling on system stability.The optimal scheduling scheme obtained by intelligent algorithms will not be suitable for the latest data stream environment because it takes too long.In order to solve the above problems,we propose ERMS(Elasticity Resource Management Strategy)based on the Storm real-time computing platform by analyzing the data flow characteristics in the big data streaming computing scenario and the Storm online resource scheduling mechanism.This strategy adjusts resource allocation in real time based on changes in online data flows.The main contents include the following three aspects:Firstly,according to the multi-group performance evaluation experiments,the main factors of the data processing delay are determined.By researching the quantification of the task topology graph node and the inter-node traffic,the performance model of the Storm elastic resource adjustment is constructed.According to the scheduling model,the concept of critical path is proposed to reduce the communication overhead in data processing by assigning strongly dependent nodes to the same working process.Secondly,the elastic resource adjustment strategy ERMS is proposed to improve the Storm resource scheduling in terms of data processing delay and system resource cost.In terms of data processing delay,by using a dynamic programming algorithm,when allocating resources to a computing node in the task topology map,all upstream nodes are optimally allocated,and then each step is optimized and reaches the overall optimality in the iterative process.The resource allocation scheme with the smallest data processing delay is obtained.In terms of system resource usage,the greedy strategy is added on the basis of dynamic programming,the available resources are gradually increased,and the dynamic allocation algorithm is used to find the optimal allocation,calculating the most resource-saving allocation scheme within the specified data processing delay range.Finally,performance evaluation of ERMS is performed from three aspects: data processing delay,throughput and system resource usage.The experimental results show that ERMS is suitable for stable data flow and elastic data flow environment.It can calculate the optimal resource allocation scheme according to the current data flow rate,effectively reduce the data processing delay of the task topology,improve system throughput,and save about 20% of system resources.