Research And Design Of Web Application Vulnerability Scanner Based On Crawler And Distributed Technology

In recent years,with the rapid development of computer technology,the Internet has become an indispensable part of people's life.As an important foundation platform,Internet has given birth to a number of powerful Web applications,involving financial,social,shopping,search,education and many other aspects.To achieve the core business,and to meet the needs of users,Web applications usually access to the internal computer system,and transfer privacy and highly sensitive information between servers and browsers,however these information has not been properly protected.Frequent Web application layer security incidents prove that the security of Web applications is far from enough.In addition,the development and appliance of new technologies,such as Web2.0,HTML5,AJAX,also bring new security threats to Web application.How to ensure the security of Web application has become a top priority.Web application vulnerability scanning technology is a kind of technical means to check the security problems of Web application from the attacker's view,it can take the initiative actions to find the vulnerabilities hidden in Web applications,eliminating danger before it happens.The research on vulnerability scanning technology is of great significance.This paper designs a Web application vulnerability scanning tool based on crawler and distributed technology,aiming at the problems that traditional scanners are insufficient in supporting JavaScript,unable to cope with the increasing scale of Web applications and the increasingly complex network environment,scanning process is blind and lack of pertinence.This paper embeds Webkit into crawler to supports JavaScript resolution and to provide a wider range of scan points for vulnerability scanners,extends the crawler and scanner to the distributed architecture to expand computing resources and improve its stability and the adaptability to complex network environment,proposes a vulnerability detection method based on state diagram to optimizes logic and improve the scanning efficiency.The main work of this paper is summarized as follow:1.In view of traditional crawler do not support JavaScript,this paper proposes a method of embedding Webkit to parse JavaScript,extract URL and dynamic input points by analysis page content and proxy requests,provide more comprehensive web site information for the scanner.2.Research on distributed technology,aiming at the large amount of data and complex network environment of website,this paper extends the scanner to distributed architecture and proposes a journaling task scheduling method based on the nodes load and communication status,which can divide and schedule tasks dynamically,ensure inter-task node load balancing,and support scanning exception handling.3.Study on the principle,attack methods and defense method of the common Web application vulnerabilities,study on the vulnerability scanning technology,summarize the deficiency of existing vulnerability scanners,and proposes a scan method based on the scanning state diagram,designe specific detection scheme for SQL injection and XSS,optimize the scan logic and improve the scanning efficiency to a certain degree.4.Based on the above ideas,complete the integral structure and the module partition design of the vulnerability scanner.Complete design and realization of each function module inside center node and scanner node.Set up test environment to test and evaluate the URL discovery ability,vulnerability scanning capability,load balancing,exception handling and other apects of the scanner,verify the availability and advantages of the design scheme.