A Web Application Vulnerability Detection Method Based On Web Crawler Technology

With the advent of Web2.0 era, Web technology high-speed developed, from the original site gradually developed into a dynamic page with a variety of powerful features.People can easily complete a lot of business through the website.Howevera lot of personal information exposed on the Internet, the corresponding security issues increased year by year.Cross-site scripting attacks are one of many security problems. In the application security OWASP 2015 China Forum, cross-site scripting attacks are a serious threat to the security of Web applications.For serious harm caused by cross-site scripting attacks, internal and external security researchers have proposed black box and white box detection methods, which can ease the harm to some extent.But with the updated Web technology, Web features supported by continued expansion, cross-site scripting attacks are constantly found to bypass security detection and filtering methods. To solve the above problem, this paper proposes a crawler-based technology Web application vulnerability detection methods, improved reptiles crawling pages, page parsing methods, and to study the vulnerability mining policy, thus solvingthe efficiency of web crawling and the ability of web analytics, improved detection of vulnerability detection rate and reduce the vulnerability of the false negative rate and false alarm rate, and to do their experiments, the results prove that the study protocol has better performance and accuracy.The paper mainly includes the following works:(1) Study the cause of system vulnerabilities andtechnologies of vulnerabilities mining, and analyzes the main means of cross-site scripting attacks, detection methods and current research status, meanwhile the current major open source crawler technology was also conducted in-depth study and learn.(2) Analyzes the page crawling and parsing the specific process, and hasresearch on searching URL and similarities to the weight based on the research of Scrapy framework. Design and implement a crawler-based Libra Scrapy frameworkto solve the efficiency of crawling and parsing page pages performance of the system. The Libra can overcome some current open-source technologies? shortcoming, such as low efficiency and poor web analysis abilities.(3) Based on the basic XSS code and different attack method, the attack code are deformed depending on some mining polices to increase the cross-site scripting vulnerability detection rate and reduce the false negative rate and false alarm rate. The main reason of deformation is to overcome the shortage of present detection methods which have single code and stored in the database heavily.(4) Implements a detection system of Web application vulnerability based on a crawler technology, which has been tested and analyzed.