Perceptual Computing Of Human Pose And Gesture Based On Deep Learning

With the ubiquitous growth of computing resources and the increased demand of consumer electronics,perceptual computing have become an active research topic due to its advantages of advantages ofnaturality,spontaneousness and adaptability.This thesis presents a systematic study on two typical tasks of perceptual computing:3D human pose recovery and gesture recog-nition.The increased demand of high precision and ease of use have posed four problems in percep-tual computing:lack of prior knowledge representation,accuracy and delay tradeoffs,cumber-some calibration processes,and lack of labeled data.Firstly,it's hard to directly embed the prior knowledge into deep-learning-based perceptual computing due to its end-to-end nature.Sec-ondly,existing sEMG-based gesture recognition methods are difficult to simultaneously achieve high accuracy and low latency.Thirdly,existing sEMG-based gesture recognition methods usu-ally require user to perform rigorous calibration every time the device is worn.Finally,existing sEMG-based gesture recognition methods usually relies on a large number of training data la-beled with gesture tags which are difficult to obtain.This thesis proposes deep-learning-based methods to address these problems.The contributions are summarized as follows.(1)An additional built-in knowledge,namely height-map,is introduced into the algorithmic scheme of reconstructing the 3D human pose under a single-view calibrated camera.The RGB image and its calculated height-map are combined to detect the landmarks of 2D joints with a dual-stream deep convolutional network.A new objective function is formulated to estimate 3D pose from the detected 2D joints in the monocular image sequence,which reinforces the temporal coherence constraints on both the camera and 3D poses.(2)A new finding is presented.The patterns inside the instantaneous values of high-density sEMG enables gesture recognition to be performed merely with sEMG signals at a specific instant.A new concept of sEMG image is introduced to verify our findings from a compu-tational perspective with experiments on gesture recognition based on sEMG images with a classification scheme of a deep convolutional network.This method enables accurate gesture recognition with low observational latency.(3)A deep-learning-based domain adaptation framework is proposed to enhance sEMG-based inter-session gesture recognition.When applied to new sessions or users,the adaptation starts working after the device is worn,and never stops until the user removes the device,going through the entire process of interaction.(4)A deep-learning-based semi-supervised framework is proposed to learn feature representa-tion with unlabeled sEMG signals and/or synchronized Cyberglove data.This method en-hances gesture recognition when only a small amount of labaled data are presented.