| Simhash is a rapid descending dimension algorithm.It can compare out the similarity between data in a short time and is now one of the most popular algorithm in mass data de-duplication or clustering.However,when various new data mining application comes out,such as Internet News Originality Recognition,or Picture De-duplication,these new application requires a higher standard on clustering,makes it hard to satisfy for troditional Simhash algorithm.This paper aims at the problem of insufficient accuracy rate of clustering in factory data such as Internet news data,when calculate the Hamming distance this paper considerd neighbor relationship between digital signature,by introducing Similarity transfer in signature,cluster accuracy was improved.When calculate Hamming distance,traditional algorithm thinks that if one of the k parts of signature is the same as the other one,then the data it represents is near-duplicate.By introducing the similarity transfer,beyond the near-duplicated data traditional algorithm judges,the near-duplicate data of near-duplicate data will also becomes near-duplicate.Because of the density-based data type of Internet news,the Hamming distance between group data is small but great between groups,so this kind of similarity transfer will increase the probability of mergings but largely reduce the probaility of one kind of data been divided into multiples.Though experiments,if slices count k is well choiced by pre-data,improved algorthm will have better clustering accuracy in density-based data such as Internet news.The experiment proves that,improved algorithm can have better clustering accuracy compared to tradition Simhash algorithm in density-based data model from 97.6% to 98.8%,the misclassification rate can be reduced by 40%,and have good application result in Internet News clustering. |
