Research On Virtual Human Sign Language Translation Based On Speech Recognition

There are many deaf-mute people in China.The number of students in urgent need of various types of education is increasing day by day.Speech to sign language translation is of great significance for enhancing the social integration of deaf-mute people,improving the teaching environment of higher education for deaf-mute students.The special translation of speech to sign language is studied,and a novel virtual human sign language translation method is proposed.The current translation methods have the disadvantages of error accumulation,and they are easily affected by the text segmentation.Hence,a one-step translation method from speech to sign language code is proposed,which can solve these problems.This method directly outputs sign language codes in speech recognition,and uses the code to perform sign language.It abandons the process of gradual translation and gets rid of the influence of word segmentation.In order to achieve the one-step translation mode from speech to sign language code,an acoustic model based on ECA-CNN-CTC and a language model for sign language code based on self-attention mechanism are proposed.The combination completes the process from speech to sign language code.The ECA attention mechanism module is used to compress the global channel features,so that the convolution approximates the global receptive field and extracts sufficient context information.The traditional convolutional neural network model is limited by the size of the convolution window and unable to obtain enough context information.On this basis,combined with CTC,an acoustic model based on ECA-CNN-CTC is constructed to achieve the mapping from speech to bopomofo sequence.At the same time,in order to meet the mapping requirements of bopomofo to sign language code,the self-attention mechanism structure is adopted to build the language model of bopomofo to sign language code.Most of the existing sign language output methods conduct through pictures or videos which are lack of reality.In particular,the method of video synthesis is not only difficult to modify,but also incoherent.The method of using virtual reality technology often takes words and characters as the basis of sign language animation production.It is difficult to complete a large-scale sign language animation library.Therefore,in response to the above problems,a virtual human sign language output method based on basic action synthesis is proposed.Also,a sign language animation library is constructed.After speech recognition,the sign language code calls each base action to form the complete sign language.This method is more intuitive.It improves the reuse rate of sign language animations and reduces the pressure of making these animations.In general,the translation method proposed in this paper is convenient and effective.It is a highly practical sign language translation way.