Research And Implementation Of Sign Language Recognition Method Based On Transformer

Sign language is the main way for deaf and mute people to express themselves and communicate,but it is difficult for those who have never learned sign language to understand it.The automatic recognition of sign language through computer vision technology can effectively promote daily communication between deaf and normal individuals,and has important practical significance and theoretical research value.Sign language recognition tasks are mainly divided into isolated word recognition and continuous sign language recognition.This article aims to study the continuous sign language recognition technology based on the Transformer model and design and implement a multi-platform intelligent sign language recognition system.The main research content and contributions are as follows:(1)In order to solve the problem of Transformer’s inaccurate extraction of local feature information in continuous sign language recognition tasks,which makes it difficult to capture hand details,a continuous sign language recognition method based on CM(Convolutional Multilayer perceptron)-Transformer is proposed.Combining the structural consistency advantage of the convolution layer with the global modeling performance of the self attention model encoder,the self attention model feedforward layer is replaced by a multi-layer perceptron;Use random frame discarding and random gradient stopping techniques to reduce the amount of training computation in time and space and prevent overfitting;A lightweight network with efficient computation was constructed,and the input and output sequences were aligned using a CTC decoder to obtain the final recognition results.The experimental results on two large benchmark datasets demonstrate the effectiveness of the method.(2)A computer-based sign language recognition and interaction system has been designed,achieving four functions: real-time acquisition,sign language recognition,video playback,and sign language learning.Using deep learning technology to automatically recognize local sign language videos or real-time collected videos using different sign language recognition neural network models in the Py Torch framework,and displaying the recognition results;Utilizing object storage technology and Py Qt framework to achieve a visual human-machine interaction interface,facilitating algorithm testing and evaluation.(3)In response to the greater demand and more convenient operation of sign language recognition in mobile applications,a continuous sign language recognition and interaction system for mobile devices has been further designed and implemented.On the Android Studio platform,an APP integrating functions such as sign language knowledge popularization,introduction of sign language datasets,sign language learning,and real-time collection and recognition of sign language has been designed through technologies such as object storage,Ok Http framework,and Flask framework,Key technologies such as real-time collection,video transmission,model call,and result feedback on mobile devices have been breakthroughs,laying the foundation for the true implementation and application of sign language recognition technology.