Study On Gesture Language Translation System Based On Neural Network

Due to the inability to communicate with healthy people,deaf-mute people face different levels of discrimination in daily life.As part of society,deaf-mute people are indispensable and important members in building a harmonious socialist.It is necessary to solve the communication barrier between deaf-mutes and healthy people,and help them integrate into society positively.So far,the auxiliary equipment used by the deafmutes to communicate with normal people is not ideal,such as electronic artificial larynx,which lacks of precise control of vocalization time and tone.The auxiliary equipment based on vision is affected by the environment.If we can design a portable equipment with high efficiency and accuracy,that would be of great practical value.With the development of the Internet of Things and Mobile Internet,wearable devices have completed the transition from large-scale to miniaturization,and have received much social attention.This dissertation designs a wearable and portable gesture language translation system.The hardware consists of two glove-like components,which use sensors to collect data.The built-in chip uses data to calculate the result of the gesture based on the pre-trained network,and the result is transmitted to the mobile terminal by the communication module.The equipment uses the bend sensor and MPU6050 chip to have a real-time observation on how the users' fingers bend and the orientation of the hand.STM32F103VET6 chip as the main control processor is in charge of calculating the result.The HC-05 bluetooth module transmit the result to the mobile terminal.At last,the result is broadcast in supporting APP in voice and text.This dissertation designs a recognition algorithm based on neural network.Firstly,this dissertation uses the equipment gather a large number of samples.Then it uses samples to train the neural networks on the PC terminal.Finally it transplants the trained weights to the data gloves.The advantage of this approach is that users can use the multicore processors to finish the complex calculation such as loop iterations in a short time.The data of the action is divided into different parts which will be trained in different neural networks.The approach ensures the high accuracy of the recognition result and also guarantees the real-time performance.The whole process is a combination of hardware and software.Besides,this dissertation tests the device's performance on individual action,successional actions and unknown actions.The reliability,timeliness and generality of the device are proved.