| As one of the most important tasks in Chinese natural language processing,Chinese word segmentation(CWS)is a primary preprocessing step for many natural language processing tasks.The result of Chinese word segmentation will directly and irreversibly affect the performance of subsequent tasks such as named entity recognition.Recently,the research work of Chinese word segmentation mostly relies on the deep pre-trained language model as the text encoder and uses the large scale pre-segmented textual dataset as the training data to achieve competitive word segmentation performance.However,the deep pre-trained language model has millions of parameters,slow inference speed,and needs enough hardware support.It is difficult to deploy in hardware devices such as mobile phones,and cannot be applied in a realistic Chinese word segmentation system.Meanwhile,in face-to-face dialogues,news broadcast and other scenarios containing multi-modal data such as text,audio and vision,only using the text to provide the facility for word segmentation seems not a good choice.Moreover,the collection and labeling of large scale data are time-consuming and resource-intensive.To solve the above problems,this thesis carries out the research on Chinese word segmentation based on text and audio.The details of this thesis are as follows:Firstly,we propose a Chinese word segmentation approach based on word boundary classification.This method still treats Chinese word segmentation as a sequence labeling task,utilizes front location classification to directly determine whether the interval between characters is a word boundary,which turns multi classification into binary classification,thereby reducing the time complexity.Using the parameter reduced deep pre-trained language model ALBERT as the text encoder,compared with the mainstream BERT,it can encode characters with faster inference speed and better performance.Because CRF cannot be used to model character labels jointly,we redesign a weight loss function based on the sample weight update mechanism of Ada Boost algorithm.Experimental results show that this method achieves faster inference speed and higher performance than other benchmark models.Secondly,we propose a Chinese word segmentation method based on multi-modal timedependent interaction.This approach is different from the previous works using textual features as input,acoustic features are used as supplementary data to integrate multi-modal information for word segmentation labeling.The encoder based on Transformer is used for uni-modal interactions of text and audio respectively.In order to better conduct the cross-modal semantic interaction,a multi-modal time-dependent interaction module is proposed.A long short-term hybrid memory gating block is designed to capture the cross-modal interaction of text and audio,so as to obtain the multi-modal character-level representation that integrates the textual and acoustic semantics.Then we use CRF to model labels jointly to perform conditional sequence labeling.Experimental results show that this approach can effectively integrate textual and acoustic features,generate the better multi-modal character representation,and obtain better word segmentation results.Finally,we propose a Chinese word segmentation approach based on multi-modal active learning.This method still focuses on textual and acoustic modalities.Only extracting the MFCC as acoustic features to further reduce the work of acoustic feature engineering.Using BERT and Res Net to perform uni-modal interactions of text and audio respectively.In order to further alleviate the work of data collection,the heterogeneous graph attention neural network is used to extract the features of dependency syntactic graph,and using the attention mechanism to fuse the features of dependency syntactic graph and the textual features to obtain the syntactic text features containing rich dependency syntactic information.Then using the multi-modal interaction module to integrate the syntactic text features and acoustic features for generating the multi-modal character representation.The linear attention mechanism is used to design a tiny long short-term hybrid memory gating block to reduce the memory consumption of cross-modal interaction.Next,using CRF as decoder to predict the output sequence.Meanwhile,integrating the active learning strategy,we propose a diversity prediction module using sentence-level multi-modal representation to predict the amount of information in sentences,and select the unlabeled samples that can represent the whole distribution of the unlabeled pool for manual labeling,while minimizing the budget for annotation.In order to reduce the computational cost of learning the diversity prediction module as much as possible,we jointly learned it with other modules.Experimental results show that this approach can effectively fuse syntactic,textual and acoustic information,and reduce the annotation effort with the integration of active learning.Compared with other active learning benchmark models,this approach can achieve better performance. |
