Research On Sign Language-to-Mandarin/Tibetan Emotional Speech Conversion By Combining Facial Expression Recognition

At present,machine learning methods have been widely used in various fields.Sign language recognition technology,facial expression recognition technology and emotional speech synthesis technology have been well developed.However,most of the existing researches mainly focus on sign language recognition,facial expression recognition and emotional speech synthesis individually.Although sign language to speech conversion has studied.But the converted speech can not express emotional information.When deaf-mute people communicate with healthy people,they often understand ambiguity because of lack of emotional expression.The thesis proposes a method of sign language-to-emotional speech conversion by combining facial expression recognition to solve the communication problems between healthy people and speech disorders.Firstly,the Deep Belief Network(DBN)method and the Deep Neural Network(DNN)method are used to obtain the predefined sign language features in the two sign language groups.The DNN method is used to obtain the facial expression features.Secondly,the support vector machine(SVM)is used to classify and obtain the text of sign language and the corresponding emotion labels respectively.At the same time,an emotional speech synthesis platform was designed by using the Hidden Markov Model(HMM)method and the DNN method.Finally,the Mandarin or Tibetan emotional speech is synthesized from the recognized text of sign language and emotional tags.The main works and originalities of the thesis are as follows:Firstly,two kinds of sign language corpus are identified and the context-dependent labels of sign language are obtained.The sign language features are firstly extracted from the 30 kinds of alphabet Chinese sign languages by using the DBN model,and a SVM classifier is used to classify the sign language with the extracted features.The kinds of 36 American sign languages are extracted features by the DNN model,and the SVM is used to classify the types of static sign language.Then,a sign language category is obtained by sign language recognition.The sign language dictionary based on the meaning of the sign language types have designed,which gives the semantic text corresponding to each sign language.The sign language dictionary is searched to obtain the text of recognized sign language.Finally,text analysis is performed on the text of sign language to obtain the context-dependent label.A six level context-dependent label format is designed by taking into account the contextual features of unit,syllable,word,prosodic word,phrase and sentence.Secondly,two kinds of facial expression corpus are recognized separately and emotion tags are obtained.The features of the extended Cohn-Kanade database(CK+)and the Japanese female facial expression(JAFFE)database are firstly obtained by a DNN model with the SVM is used to classify.Then,the emotion tags are obtained by facial expressionrecognition to select the emotional acoustic model for synthesizing emotional speech.Finally,the thesis realizes sign language to emotional speech synthesis.The HMM-based method is firstly used to train the acoustic model,meanwhile,the speaker adaptive transform method is used to realize the emotional speech synthesis.The DNN-based speaker adaptive method is then used to obtain the speech model of target emotional,and achieved the synthesis of target emotional speech.The DNN is finally used to train the acoustic model,and then the speaker adaptive(DNN)method is used to realize the Mandarin-Tibetan bilingual emotional speech synthesis.