Research And Design Of Distributed Web Vulnerability Scanning System Based On MapReduce

In recent years,Web applications have been attracting people's attention because of their short development cycle and easy maintenance,and have become a highly interactive and feature-rich interactive media.Web applications bring great convenience to people,but also bring a lot of security issues,such as many illegal people through the Web application vulnerabilities damage other people's servers,stealing privacy and business data so that users suffered a huge loss.Loopholes can not be directly observed,must be a series of detection methods to accurately identify.If you can scan the Web vulnerability in advance,to find loopholes,and then repair loopholes or take defensive measures,it can effectively prevent hacker attacks.Therefore,Web vulnerability scanning system as the first protection against Web attacks,has been widely concerned about and use.With the passage of time,the traditional single-plane vulnerability scanning system has gradually exposed a large number of problems,such as scanning efficiency,scope,result analysis and so on are obviously insufficient,can not adapt to the current stringent safety requirements.To solve this problem,this paper designs a distributed Web vulnerability scanning system based on MapReduce.In this paper,the Web security vulnerabilities and the reasons for the emergence of a specific analysis,and based on the Web crawler technology and MapReduce distributed computing framework,the two match with a new program.Then,a task scheduling algorithm T-STSA based on priority is designed and implemented.The task is dispatched to the virtual machine which conforms to the objective function,and the task scheduling strategy is proved experimentally.Then,this paper divides the system into two basic modules: Web interaction,task scheduling,scanning engine,distributed crawler and plug-in library.This paper focuses on the detailed design of each module.The Web-based interactive module provides a visual interface for users.The task scheduling module effectively improves the efficiency of vulnerability scanning through the scan task processing mechanism based on MapReduce and the T-STSA scanning task scheduling algorithm.The scanning engine module adopts stateless technology and semi-connection The distributed crawler module provides the basic web crawler functionality for the system.The plug-in library module can save different scan plug-ins,making scanning more flexible and thorough.After the system is deployed,the distributed Web vulnerability scanning system is tested and tested.Through the analysis of the test results,the system works well and realizes the functional requirements and improves the scanning efficiency.