| According to the statistics of ACM, the number of repeated web page accounts forabout30%-45%. With the increasing number of search engines and the improvement ofusers’ requirements, the search quality becomes the weight to win the users for all of thesearch engines. If the duplicated web pages removed timely, search engine can not onlysave a lot of storage space, indirectly reducing equipment procurement cost, but alsoimprove the retrieval quality of the network and accessing efficiency. Finally, itimproves satisfaction of users.The key points of the elimination of duplicated web pages are text featureextraction and the calculation of large-scale informations. Traditional text featureextraction algorithm is generally divided into three categories. The first one is based onURL which only removing the mirror site. The second one is based on the matching ofcharacter string which has high accuracy and high time complexity. The third one isbased on clustering. The last method is very high in recall, but its accuracy is relativelylow for the news and the template texts.By analyzing near-duplicated web pages, found that repeated pages may havemuch change in the content, but few document format. In view of this characteristic, thepaper puts forward two algorithms based on text structure tree.The long sentence doesn’t representative theme of the web page. Facing pagecollector change rules, the longer the sentence is more fragile. This paper puts forwardthe algorithm based on text structure tree and key words to improve the algorithm basedon long sentences. The algorithm extracts sentences which contains keywords as keysentence. And the number of features is determined by the length of paragraphs.Experiment shows that the improved algorithm effectively avoids these two drawbacks,and the accuracy and recall rate are improved.The smaller feature is hashed was less interference. According to the feature,algorithm based on text structure tree and character strings is proved. Firstly, it extractsthe head and tail words of a certain sentence in which high frequency punctuationsoccur. Secondly, it generates the fingerprint with Bloom Filter algorithm. Finally, itdetermines the similarity according to the layer fingerprint. Experiment shows that this algorithm has greatly improved in the recall rate, which is especially in small documents,and greatly reduces the time complexity. |
