Research Of Feature Vector Value Weighted Based On Semantic Analysis In Chinese Text Clustering

With the rapid development of Internet, people find it difficult to find their useful information through Internet. The application of text mining is very broad, while text clustering is an important part of text mining.Text clustering is an important research branch of clustering method and it is the application of clustering method used in text processing field. But text clustering has some common problems such as high-dimensional text object, a document collection may need hundreds of thousands of words to express; the sparse of text object, a lot of words are rarely used. However the literal meanings of text object and potential semantic relations could hardly be excavated.First, this paper reviews some concept and methods of Chinese text pre-processing, text representation, text similarity calculation, and document feature vector reduction, clustering algorithms and clustering efficiency evaluation. Then this paper discusses the main problem of text clustering in excavation of literal meaning of text object and semantic relations and presents several methods of feature vector value weighted based on semantic analysis: based on word part of speech, based on the term location, based on term length, based on the term correlation and based on term similarity.The term correlation means the probability of two terms appearing in a certain language environment. In general we use document co-occurrence frequency, paragraph co-occurrence frequency and sentence co-occurrence frequency to measure term correlation. Study shows that the method of paragraph co-occurrence frequency based feature value weighted can improve the text clustering result by 10%.The term similarity means the ability of replacement of two terms in different language environment without changing their syntax structure. In this paper, semantic similarity is calculated based on "HowNet". Study shows that the method of semantic similarity based feature value weighted can improve the final clustering effect. But it is worse than the method of term correlation based. This paper will also try to find some reasons to explain this difference.Finally, this paper discusses the possibility to combine these semantic analysis based value weighted methods. Study shows that the combination of these methods is better than anyone of them.