Research On Computer Vision Technologies For Human-robot Motion Retargeting Based On RGB-D Information

Nowadays,to let the service robots perform complex tasks under dynamic conditions,ensuring the flexibility of the operation and also their motion fluency remains a hot topic in robotics.In this dissertation,I will focus on how to generate natural and smooth motions for the robots to help them survive in complex tasks.The existing approaches can be classified into planning-based or imitation-based methods,among which the human-robot motion retargeting has received great attention as the input is quite simple and the motion is natural to human.The goal of human-robot motion retargeting is to let a robot follow the movements performed by a human subject.Previously,human motion estimation is achieved with motion capture system that tracks the markers attached on the human subjects.The availability of low-cost commodity RGB-D sensors together with accurate human estimation,has made the human-robot motion retargeting easier than ever.However,the retargeting problem still suffers from the insufficient depth accuracy,complicated 3D human modeling,different predefined mapping for different robots,etc.To address these problems,in this dissertation I have focused on several key technologies for human-robot motion retargeting with a single RGB-D sensor.Particularly,I have proposed methods for real-time artifact compensation for depth images acquired from multi-frequency Time-of-Flights(To F),dynamic non-rigid objects reconstruction with RGB-D information and human-robot motion retargeting approach using RGB-D sensor:1)First,the captured depth from To F sensor has significant artifacts under dynamic scenes with fast motion as caused by its working mechanism of phase fusion.To tackle this,I propose an efficient motion compensation method to suppress the motion artifacts by combining image flow and kernel density estimation for multi-frequency unwrapping.Firstly,we capture the raw multi-phase images and calculate the optical flow between every frequency.Secondly,we generate multiple depth hypotheses and uses a spatial kernel density estimation to rank them with wrapped phase images.Finally,we can get the accurate depth from fused phase image.We have validated our algorithm on Kinect v2 and used GPU to optimize the pixel-wise part.The method shows its effectiveness on real datasets with real-time performance.2)Next,for the preparation of human-robot motion retargeting task and get more accurate motion estimation,I also put effort on 3D human body modeling from a single RGBD sensor,which is a challenging task as we consider the almost inevitable accumulation error issue in previous sequential fusion methods and also the possible failure of surface tracking in a long sequence.To deal with this problem,I propose a global non-rigid registration framework and tackle the drifting problem via an explicit loop closure.Our novel scheme starts with a fusion step to get multiple partial scans from the input sequence,followed by a pairwise non-rigid registration and loop detection step to obtain correspondences between neighboring partial pieces and those pieces that form a loop.Then we perform a global registration procedure to align all those pieces together into a consistent canonical space as guided by those matches that we have established.Finally,our proposed model-update step helps fixing potential misalignments that still exist after the global registration.Both geometric and appearance constraints are enforced during our alignment.Experiments on both synthetic and various real datasets have demonstrated the capability of our approach to reconstruct complete and watertight deformable objects.3)Finally,I present a novel approach for human-robot motion retargeting that combines the human pose estimation and motion retargeting procedure in a unified generative framework.A 3D parametric human-robot model(HUMROB)is proposed that has the specific joint and stability configurations as a robot while its shape resembles a human subject.We use depth map as input and drive the HUMROB to fit the input 3D point cloud.The calculated joint angles of the fitted model can be applied onto the robots for retargeting.The robot’s joint angles,instead of fitted individually,are fitted globally so that the transformed surface shape is as consistent as possible to the input point cloud.The robot configurations including its skeleton proportion,joint limitation,and Do F are enforced implicitly in the formulation.The proposed framework is tested with both simulations and on real robots that have different skeleton proportion and Do Fs compared with human to show its effectiveness for motion retargeting.4)In addition,I exploit the RGB image in our algorithm to initiate the overall pipeline.In details,we predict the 2D pose with deep convolution network,then get the 3D joint position with its corresponding depth map.The head tracking is specified particularly to improve the performance.Besides,instead of relying on reconstructed human models,I take advantage of a probabilistic human template to generate a personalized and drivable human model for the human subject only from a single shot,which is more effective and convenient and we have shown that it is sufficient for the retargeting purpose.