| Recently,there are more than 20.54 million deaf people in China,sign language is the most frequently used expressions in daily communication between deaf people.In the past,sign language teaching mainly relied on manual teaching and video teaching.Manual teaching is difficult to give the teaching demonstration at any time,anywhere and repeatedly due to limited manpower,while video teaching lacks timely correction of wrong sign language actions.In dynamic sign language sequences,there are often subtle differences of hand gesture or movement direction,which are easy to be ignored in the learning process and lead to wrong meanings.Therefore,there is an urgent need for sign language teaching equipment to identify correctness of sign language made by students.Based on deep learning,this thesis studies the standard sign language teaching technology by the combination of virtual and physical robot.It includes hand position detection,sign language correctness discrimination and virtual and physical robot sign language teaching.Video object detection has the characteristics of large number frames and strong relation between adjacent frames.This thesis designed improved Faster RCNN+FPN to detect the hand at key frames,it enlarged the FPN features to the size of the C2layer for superposition.Designed FGSFP+TFFR model for hand detection at non key frames.FGSFP accelerates the network and reduces the computational by using flow guided features and the designed method of obtaining proposals by stable and flow key frame detection results.TFFR is proposed to adjust the forward optical flow to obtain a more accurate center point offset refine map.TFFR solves the inconsistency between the forward optical flow and the box center point offset.In general sign language recognition,only sign language categories are recognized,and there is no standardization discrimination of the sign language actions.At the same time,there are some problems in continuous sign language recognition,such as excessive action influence recognition accuracy,low recognition accuracy of continuous multiple sign languages.To solve the above problems,this thesis implements DCSR3D+GRU encoder-decoder network to make a comprehensive correctness discrimination on the category and standardization of sign language actions.The left and right hand detection boxes obtained in the hand detection phase are used to get the hand patches.The hand patches are encoded by DCSR3D.DCSR3D adds 2D deformable convolution to the residual unit in R3D to enhance the extraction of current sequence features.It concat 4 different paths of 3D convolution and 2D deformable convolution feature results,and adds sequence attention mechanism inside the residual block.The hand feature encoded by DCSR3D is concat with the embedding of the sign language text to be learned as the input of the GRU decoder.The final output is the sign language category and standardization comprehensive correctness discrimination result.In this thesis,the sign language correctness discrimination dataset is collected.Each sign language video has two kind labels of sign language category and standardization category at the same time.The improved Faster RCNN+FPN semi supervised learning method is used to make the hand position labels in the video.In this thesis,a sign language teaching system with the combination of virtual and physical robot is developed.The teaching system is controlled by the UI interface of Unity3D,the virtual teacher model and sign language animation are made by Maya software,the physical robot is controlled by PC serial port.Embed video hand detection model and sign language correctness discrimination model into Unity3D.The system provides sign language action demonstration and situational interaction through virtual teacher and physical robot.It can also record sign language practice videos of learners,and use deep learning method to identify the correctness of the sign language actions. |
