Research Of Gesture Recognition And System Design Of Gesture Interaction In The First Perspective

With the continuous emergence of head-mounted devices such as Augmented Reality(AR)and Virtual Reality(VR),head-mounted devices have attracted much attention.Currently,the interaction mode of the head-mounted device in the market is mainly a device-controlled contact mode.However,the visual-based non-contact interaction mode can provide users with a superior interactive experience.Specifically,the vision-based gesture interaction method requires the head-mounted device to carry the camera,and collects the interaction gesture information from the first perspective.The factors such as light variation,motion blur,and gesture occlusion make the application very challenging.Therefore,how to apply the visual gesture interaction method to a head-mounted device is a very worthy research work.This article focuses on the difficulties in visual gesture interaction based on the first perspective,and the interactive application of gestures in head-mounted devices.The color camera is used as the image acquisition device,and the following main work is carried out:(1)For layer-by-layer error accumulation brought about by module combination in static gesture recognition algorithm,a method named Hand-RCNN based on Faster-RCNN network is proposed.At the same time,the three functions of the static gesture detection,recognition and fingertip positioning in the first viewing angle are realized,including:a)image preprocessing:average regularization,increasing the edge contour channel,introducing bulk normalization,and strengthening model feature expression,b)fingertip positioning:fingertip positioning from the point of return to the target detection problem,combined with multiple points to fine-tune the correction,making the fingertip positioning error is reduced to 3.77 pixels,to get a better fingertip positioning effect.c)Optimize network structure:For feature extraction network,RPN network,ROI Pooling layer and classification output,further simplify network parameters and reduce feature mapping dimensions.While ensuring network accuracy,the number of network parameters is reduced to 37M,and the calculation speed is improved to 35.71 frame per second(fps)meets the real-time requirements of gesture interaction.(2)A set of gesture interaction system from the first perspective was built for the requirements of the headset application scenario.Specifically,based on the Hand-RCNN model,motion tracking,smoothing filtering and other methods are used to implement various gesture interaction methods such as gesture roaming,confirmation,waving,and two-handed kneading,and on this basis,the Sony HMZ-T3 head is used.Wearable device as carrier,related gesture set instruction and interactive interface were designed,and a set of audio and video entertainment system with gesture interaction as the main body was built.In addition,this article also realizes the system's transplantation on the mobile platform,and uses NEON optimization technology to further optimize the performance,aiming to expand the application of gesture interaction in head-mounted devices.