Research And Implementation Of Sign Language Translation Algorithm Based On Speech Recognition

There are more than 20 million people with hearing and speech disabilities in our country.They cannot communicate with the outside world by speaking and listening like a healthy person.They can only communicate with others through silent sign language.In order to make deaf and mute people better integrate into society and communicate effectively with society,sign language translation plays a vital role.With the continuous development of artificial intelligence,various deep learning technologies continue to emerge.In view of a large number of deaf people and less research on speech to sign language translation in our country,how to integrate artificial intelligence into sign language translation,so that deaf people can easily "understand" the sound content of healthy people has broad and practical research and application value.This thesis focuses on sign language translation based on speech recognition,and the main contents are as follows:(1)Speech recognition is the basic work of speech to sign language,which can be divided into two major tasks: acoustic model and language model.This thesis first studies the acoustic model of speech recognition,improves the DFCNN(Deep Fully Convolutional Neural Network)framework proposed by iFLYTEK,proposes CNN + CTC(Convolutional Neural Network+ Connectionist Temporal Classification)algorithm,and realizes the end-to-end acoustic model.This algorithm can effectively recognize speech data as Chinese Pinyin sequence,and the word error rate and sentence error rate on the test set are 9.20% and 24.02% respectively.Compared with two common acoustic models,the recognition effect has been greatly improved.In terms of language models,since the language model algorithm in this thesis can be understood as translating Pinyin sequences into Chinese character sequences,so the text translation framework,Transformer,is applied to the language model.The language model algorithm of this thesis,based on Transformer Encoder,is realized through TensorFlow.In the same test set,the language model algorithm proposed is better than the two commonly used language models in overall effect.(2)After the speech recognition algorithm recognizes the speech as text,a Chinese word segmentation algorithm is required to segment the sentence.In this thesis,the MMSEG Chinese word segmentation algorithm based dictionary and the Bi-LSTM(Bi-directional Long Short-Term Memory)Chinese word segmentation algorithm based on statistical sequence labeling are implemented.The two segmentation algorithms are tested separately.According to the experimental results and the sign language translation system requirements,the Bi-LSTM Chinese word segmentation algorithm is selected as The Chinese word segmentation algorithm in this thesis.(3)A sign language picture set suitable for the speech-to-sign language translation system of this article is constructed,with a total of 4,993 sign language pictures.This set is available for download.A MySQL database for sign language pictures,my_sign_picture,is set up.Sign language picture labels,sign language picture paths and image binary streams are saved to the sign_picture table.Sign language pictures can be easily and quickly searched through SQL statements.(4)This thesis builds local and Web speech-to-sign language translation systems.The speech recognition algorithm,Chinese word segmentation algorithm,and sign language picture query are applied to the systems,which can adapt the speech-to-sign language translation tasks under different needs,and facilitate deaf and mute people to communicate with the outside world.