Design And Implementation Of Distributed Stress Test System Based On Spark Load Balancing

With the development of modern Internet technology,software integration and user scale continue to expand,software and e-commerce sites such as video portals,12306,etc.need to withstand the concurrent access of a large number of users,stress testing has become an indispensable key step before going online.In the face of increasing pressure testing needs,the commonly used stress testing frameworks are no longer able to effectively support the diverse needs of the industry.This paper analyzes and discovers the problem defects in the existing commonly used stress test software,such as: difficult configuration,low real-time,unbalanced load and no support for multiple services.In response to these problems,this paper proposes the design goals of the improved system,and develops a distributed stress testing software system based on Spark load balancing.The system is mainly composed of control management,stress test engine,Yarn load adjustment,service plug-in,etc.;and provides a unified control management function,which can reduce user operation configuration and display the test results in real time.By adopting the distributed stress test engine designed by Spark Streaming,it can quickly generate distributed high concurrency pressure and improve multi-threading concurrency efficiency.And through the flow calculation to process the test result information,providing the underlying computing support for the real-time performance of the system.By improving the resource scheduling algorithm based on Yarn,the load balancing of different node stress test engines is realized,and the overall performance of the system is improved.Service plug-in has also been designed to quickly add new compression services to the test system,with the aim of extending more different types of service support.Finally,the distributed stress test system designed in this paper is compared with the commonly used stress test system.The test results show that the system can support the node load balancing,real-time stress testing,multi-service support,etc.,as well as the practicability,efficiency and stability of the stress test by using this system.At present,the system has been officially launched in the basic service of cloud platform in iQIYI.It is responsible for the daily stress testing of several service businesses and has achieved good reputation and user feedback.