Survey On The Research Of Focused Crawler Technique Based Vulnerability Management Platform

With the development of the current network, people on the information collection requirements become higher and higher, but the traditional generic crawler technology as a means of information gathering techniques, have been unable to meet the needs of the current users. So focused crawler technology emerges, it is a new technology with a certain relevance,can use some specal rules to gather the-informarion. Studied in this paper is a focused crawler technology based vulnerability management platform, compared with the current various of crawler technology, it has its own unique features and advantages.In this paper, we describes the background and current development status of crawler technology, and by comparing the old generic crawler technology,we lead to new focused crawling technique. Then we introduce the background platform of focused crawling technique-vulnerability management platform project. By introducing the vulnerability management platform, we lead to the final core module (crawler module) and core technology (focused crawler technology).Next,we analysis the technique part-by-module, and decompose the various technical points into each sub-module for detailed research and analysis.Finally, we summarise the results of every individual sudies, get a report contain the advantages and disadvantages of the entire technique.At last, according to the analysis results and some disadvantages of current crawler technology, combine with the latest research results at home and abroad about focused crawler technology, we inprove the focused crawler technique in two ways:improve the efficiency of crawler and improve all aspects of URL.Through using the means above, we improve the overall level of focused crawler technique based vulnerability management platform.