Development Of A Wearable Sign Language Translator

With the rapid development of smart devices, there are many ways developed for communication, but speaking language is always considered as the primary means of communication between people. Hearing people are able to communicate through talking, but the deaf can only use sign language, while it is difficult for hearing people who are not familiar with sign language to communicate with them. Therefore, it is of great demand to develop a device that is able to translate sign languages into speech or text that is friendly to hearing people. Sign language recognition is such a technique for translating sign languages to other forms of expression that is straightforward for hearing people to understand. Compared to other gesture sensing technology, the approach using surface electromyography (SEMG) sensors and motion sensors has significant advantage in gesture caputure. The surface EMG can be used to directly detect muscle activity of fine geatures like finger movements, and motion sensor is able to measure large-scale trajectory during gesture implementation. Therefore, the combination of both sensors achieved a great number of applications in gestural interaction.With the basis of the pilot studies in our laboratory, a sign language recognition framework was proposed based on the information fusion of4surface EMG sensors, a3-axis accelerometer and a3-axis gyroscope plased on each arm. A device for sign language recognition was also designed and developed based on dual-core DSP. This study can be considered as the first attempt to develop sign language recognition devives toward commercial applications. It really laid a solid foundation for marketing such devices. The primariy contribution of this dissertation can be summarized as the following aspects:(1) A framework for fusing8-channel SEMG signals,6-channel acceleration (ACC) signals and6-channel angular velocity (AV) signals is proposed. First, the data are segmented based on the muscle activity measured through the4-channel SEMG on the right arm. Second, mean absolute value and coefficient of auto regressive (AR) model of SEMG and downsampling points of trajectory signals are extracted as features which will be used for pattern recognition. Third, a set of classifiers are used in a decision tree structure. The decision tree included single-/double-handed classifier, orientation classifier, and multi-stream hidden Markov model (HMM) from top to bottom to produce a final decision of the geature to be indentified.(2) A portable real-time sign language recognition device based on dual-core DSP is realized. The device consist two arm-band-like part designed to be wron on the left and right arm, each part contains4SEMG sensors, a3D accelerator and a3D gyroscope, which can be tied on muscle belly of forearm. The right arm band is the slave device which captures SEMG signal, acceleration signal and angular velocity signal from the right arm and sends them into the left one through Bluetooth. The left arm band works as a master device which not only captures the same amount of data from the left arm but also synchronize data from both arms. In addition, corresponding signal processing algorithm is performed on the left.master armband as well. Finally, the recognition result can be sent through Bluetooth and the sign can be accordingly translated into speech.(3) A computational procedure called framing HMM is realized on DSP. the high computational complexity of HMM is very likely to lead to long time delay during the recognition. By solving this issue, this paper proposed a method which can split the procedure of computing into several parts. After computation of every part, key parameters are restored into memory for future recognition after the sign gesture is completely performed. Those parameters for computation of every part can be obtained after every frame of SEMG, therefore the framing HMM observably reduces the recognition delay as compared with the previous solution that the HMM is not excuted until the sign gesture is completely performed.