| Hand gesture interaction is the key technology of “Industrial 4.0” and “Made in China 2050”.The using of hand gestures interaction in intelligent manufacturing can improve the production efficiency of intelligent manufacturing and increase the worker's working comfort.In hand gesture interaction,monocular vision-based gesture interaction can cooperate with ordinary mobile devices with cameras to achieve free interaction easily.It is the mainstream in gesture interaction at present.The existing interaction methods mostly aim at static gestures,or only deal with simple singlemodal gestures,or have too much restrictions,so they influence the availability of gesture interaction.Therefore,it is necessary to extract the multi-modal features of gestures and analyze the different factors of their continuous changes.Aiming at the space-time diversity in moving robot hand gesture interaction,this thesis starts with the video sequence captured by monocular camera under different conditions,uses the theories and methods of computer vision and machine learning comprehensively,makes theoretical analysis and experiments,and verifies the feasibility of the proposed methods in manufacturing environment.This thesis mainly focuses on key issues such as shape feature in static hand gesture,trajectory feature in dynamic hand gesture,hand gesture feature extraction in camera moving,hand gesture feature extraction when the pose of the camera are different,and feature extraction of in-air writing digit string.The specific contents include:1.As hand is a complex multi-joint non-rigid object,the palm and fingers constantly change in motion,up to 27 degrees of freedom,so it is a very difficult and complex task to analyze the high-level operation characteristics of gesture model by visual technology,such as fingertip pointing,fingertip position and so on.To solve the above problems,under simple segmentation conditions,a method of extracting image structural feature points is proposed;using the extracted structural feature points as the data,the gesture model is established and optimized to get the high-level features,which can be used for palm,finger and fingertip positioning.Experiments show that the detection rate of fingertip is 88.9%,and the proposed method can be used when accurate gesture boundary is not available.2.As the existing dynamic gesture recognition methods still have the problem of low recognition efficiency in interactive systems such as in-air writing.In order to increase the recognition efficiency of dynamic gesture trajectory recognition,by introducing the curvature angle into feature extraction,a new gesture trajectory feature extraction method is proposed here.Experimental results show that,the new trajectory feature can distinguish the trajectories with same shape but different paths and the mean classification accuracy is increased to 93.33%.3.Moving monocular camera can enlarge the observation range,but it also causes the global motion of image coordinate system,so it influences the target trajectory observation in gesture recognition.For the problem above,the target trajectory acquisition model based on homography is established and proved.Through the analysis of the mapping relation between the adjacent key-frames,the model uses the homography matrix solvation and feature points re-projection to get the relative moving vector between adjacent key-frames,at last the target trajectory of ground truth is obtained.Experiments show that,using the proposed method,in moving camera situation,the dynamic hand gesture's mean classification accuracy is increased by 25% and the processing speed is 1.47s/frame,which satisfy the requirement of application.4.When the camera's visual angle is different,the same motion trajectory will project into different trajectories,which will affect the recognition and application of the trajectory.In order to solve this problem,in situation of planar hand moving in same plane,motion based camera pose self-calibration model and view-invariant hand gesture feature extraction method are proposed,and then the model is expanded to the situation of auxiliary object based-arbitrary hand moving along plane.(1)When the planar hand moves in same plane,based on camera imaging model and feature point motion constraints,a camera pose self-calibration model is proposed;then,using the calibrated camera pose,three dimensional trajectory can be recovered;finally,through trajectory re-projection,the captured trajectory is rectified to realize view-invariant hand gesture feature extraction.(2)Based on the above camera pose self-calibration model,the constraint that feature points belonging to the same plane is changed to the constraint of feature points moving along plane,and a new camera pose self-calibration model is established to calibrate the camera pose.At last,the view-invariant feature extraction method is expanded to the situation of auxiliary object based-arbitrary hand moving along plane.Experimental results show that,after view normalization,the gesture trajectory extracted is more likely the front shot,the average recognition rate of the dynamic gesture is increased by 22%.5.In the process of gesture continuous in-air writing operation,there will exist operation gesture sequence and transition gesture sequence.How to recognize different states in interaction through visual technology,and then recognize continuous gesture strings? A gesture string feature extraction and recognition method based on user behavior analysis is proposed.From the perspective of cognitive theory,the gesture micro-behavior model is established;then,key-frames of gesture sequence are selected to divide the gesture sequence;finally,combined with the neural network and the optimization model,the gesture string with the best overall recognition rate is selected as the result.Experimental results show that the proposed method can realize the segmentation and recognition of continuous gesture strings composed of two digits,and the precision rate of a digit is 86%. |
PDF Full Text Request