Design Of Barrier-free Two-way Interactive System For Deaf-mute People With Sign Language Real-time Translation

The number of people with various kinds of disabilities in China has reached 85 million,accounting for about 6 percent of the country’s total population,according to surveys based on the national census and spot checks on the disabled.Deaf-mutes make up a huge proportion of them,more than 20 million.Sign language is a way for deaf-mutes to communicate with each other,but when they communicate with able-bodied people,they encounter various difficulties.Before this,the function of the bidirectional interaction system between the deaf and the able-bodied has not reached the ideal effect: for example,the equipment using em G signal for the interaction between the deaf and the able-bodied can only one-sided reflect the arm or upper limb neuromuscular activity,its recognition rate is not ideal;However,convolutional neural network based deaf-mute interaction equipment in complex environment,recognition rate will be affected.And the above equipment only realized the deaf-mutes to the healthy person communication process,did not realize the healthy person to the deaf mute person communication process.Therefore,this paper developed a barrier-free two-way interaction system of sign language real-time translation for the deaf-mutes.When the deaf-mutes communicate with the able-bodied people,the deaf-mutes wear data gloves and gesture with the bending sensor to collect the bending signal of the fingers of the deaf-mutes,which is transmitted to MATLAB through serial port communication with the attitude sensor on the back of hand and forearm.When MATLAB receives the posture data of the back of the hand and forearm,it normalizes the data,and then puts the posture data of the back of the hand and forearm into the neural network which has been trained in advance as input,and then the neural network determines the output.Will get the eigenvector wireless transmission via a serial port to the Arduino microcontroller,the eigenvector corresponding to conform voice makes SYN6288 Chinese speech synthesis module and gestures that corresponds to the sound,the deaf gesture information into healthy people can understand gestures,to complete the purpose of the deaf and dumb people want to communicate with the healthy people.When the able-bodied person wants to communicate with the deaf-mute person,the able-bodied person makes a sound to the LD3320 A voice module,which recognizes the sound.When the sound is recognized successfully,a hexadecimal command is sent to the Arduino microcontroller,and the Arduino microcontroller will receive 22 hexadecimal data corresponding to the human body Angle.Used to control the character in Unity to make the sign language animation corresponding to the voice.The speech information of the healthy person is transformed into gesture animation that can be understood by the deaf-mutes to complete the purpose of communication between the healthy person and the deaf-mutes.In the experiment conducted in this paper,we selected a number of volunteers for sign language and speech experiment respectively.The experimental results of sign language experiment show that the average recognition rate of sign language experiment is 94.9%,and the experimental results of speech experiment show that the average recognition rate of speech experiment is 90.8%,which verifies the effectiveness and feasibility of the bidirectional interaction system developed in this paper.