| A word is the minimum unit of Chinese language that can exit independently and meaningfully.English words can be clearly divided through Spaces,but there is no delimiter between Chinese words,and there is no clear morphological mark in the word itself.Therefore,Chinese words cannot be divided through spaces like the western language.The unique matter of Chinese information is how to divide the Chinese string into reasonable sequences,which is the Chinese words segmentation technology.However,in large text data with common themes in a certain area,the amount of data is increasing,such as some social media and website reviews.In these areas,new words often appear,which are difficult to find and include in the dictionary in time.With the emergence of new words and domain words,traditional dictionary-based Chinese words segmentation technology is difficult to solve the problem that new words can not be recognized and real-time word segmentation is slow.It is also a major difficulty in the study of Chinese words segmentation technology.The problem of this thesis is how to use big data technology and new segmentation algorithm to segment words quickly in the field of large-scale data.The system designed in this thesis adopts a domain-oriented large data fast word segmentation technology,which is based on statistical word segmentation method,and involves the field of large data and natural language preprocessing,There are two main processes.Firstly,the frequency model of adjacent words in corpus is established by using big data technology.Secondly,the new segmentation algorithm is used to find the appropriate location and divide the sentences to be segmented into several word segments.The system is mainly composed of three modules:building mapping table,Chinese word segmentation service and result display.The mapping table is constructed by the Spark cluster,the front-end display is designed with Vue and HTML+CSS,and the back-end word segmentation is done by the Flask server.The fast word segmentation methods and effects adopted in this system have the following characteristics:(1)based on statistics,no need to build dictionaries in advance,and other manual intervention;(2)strong ability to discover new words in corpus of special domain;(3)the time complexity of word segmentation is O(n),in specific scenarios,the time of word segmentation is faster than other dictionary-based word segmentation tools,such as Jieba segmentation and thulac segmentation. |
