| Chinese word segmentation is the pre-processing stage of Chinese information processing, and its task is to use machinery method to automate the identification of word boundary in Chinese texts. The few key issues in Chinese word segmentation are: Corpus construction, segmentation disambiguation, and out-of-vocabulary word recognition. This paper focuses on disambiguation, and to solve the feedback problem of the segmentation system, a rule-based statistical word segmentation algorithm is proposed.Dictionary-based word auto match is the first stage of word segmentation, and the structure of the dictionary has a big impact on the matching speed. For the dictionary based on first character hash or the first two character hash will inevitably need to binary search the rest characters of the word, thus decrease the matching speed, we propose a full-hash prefix tree structure. The time complexity of word matching is reduced to O (N), where N is the average length of the words. Experiments show that the structure is at least 4 times faster than first character hash structure in word matching.The statistical disambiguation method is greatly affected by the quality and the balance of the corpus; they can not cover some special cases. This paper introduces the concept of segmentation rules. To solve the wrong segmentation situation, this method allows the user to write segmentation rules to give the segmentation system some feedbacks, thus correct the mistake. We also integrate the word matching process, the rule matching process, and the statistical disambiguation process in a unified algorithm. This rule-based statistical word segmentation algorithm has a time complexity of O (M), where M is the length of the sentence to be segmented. The speed is almost the same as the two-way matching word segmentation algorithm. Experiments show that the segmentation rules enable the segmentation system to have the feedback mechanism and the mechanism of accumulation of knowledge. This is a useful complement to statistical models.Finally, we develop the rule-based statistical word segmentation system. We use the corpus from the State Language Commission to test the system, and the results show that the segmentation accuracy is 96.3% or more, and the recall rate is 96.7% or more. Users can write segmentation rules and add word entry to the dictionary, thus effectively improving the segmentation performance of the system. |
