Research On Chinese Sign Language Detection And Recognition Based On Feature Fusion

As an important part of Chinese sign language,Chinese finger language is also an important position in the recognition of sign language.Finger language as a communication tool for the hearing impaired impairment,makes it more convenient for them to interact with the outside world and man-machine information.Traditional sign language recognition requires high gesture features.In order to solve the detection and recognition of Chinese sign language under complex actions,this article starts from the target detection and recognition method of dynamic sign language.The main research contents include:(1)Improved an RCNN network based on Feature Pyramid Network and Attention Mechanism to detect Chinese sign language targets(abbreviated as FPN-AM-RCNN).First,use the Res Net50 network to extract the features of sign language images,and then replace the RPN part in Faster RCNN with the Feature Pyramid Network(FPN).Because FPN has the feature of multi-scale extraction of features,The extracted features are richer;in the ROI area of interest The attention mechanism is introduced to focus attention on the part of interest,thereby increasing the speed of model formation.Finally,the extracted features are classified and regressed.By testing on the Chinese sign language dataset,comparing the improved FPN-AMRCNN method with the classic Faster RCNN network,the results verify that this method has a greater improvement in the detection of small and medium sign language targets.(2)Due to the low accuracy of the SSD model for the recognition of small and medium targets,a Chinese sign language recognition method based on the SSD(Single Shot Multi Box Detector)-Transpose convolution feature fusion based on regional sampling optimization is proposed.Regional sampling SSD-Transpose fusion method strategy steps: First,it is necessary to optimize the low-level sign language feature layer of the SSD model to improve the recognition and detection speed of small and medium targets.Secondly,perform the deconvolution operation on the features extracted from the SDD model.The features of small and medium sign language targets are merged to improve the accuracy of the recognition of small and medium sign language targets.Finally,we test on the Chinese sign language dataset.Experimental results show that the improved SSD-Transpose method greatly improves the ability to identify and detect SSDs.Small and medium sign language goals and achieved good recognition accuracy.(3)A dynamic sign language recognition system is designed and implemented,which mainly includes five parts: human-computer interaction,image preprocessing,sign language detection and result display.Human-computer interaction The module provides an interface for querying images(images to be retrieved)and query ranges.Before the retrieval program is executed,the dynamic sign language needs to be cut into pictures at a preset frame rate,and they are named with time clues according to the frame rate.Realize the traceability of the image,that is,locate the moment of its appearance in the entire action according to the name.The sign language retrieval module uses the Res Net50 restore Algorithm for retrieving the sign language image of the same object as the query image in the sign language image The database input by the sign language detection module.