Dynamic Human Body Projection Mapping Based On Kinect

Projection mapping,sometimes called spatial augmented reality,is the technique to alter appearance of real world objects by projection.Projection mapping can be used to augment arbitrary surface without the need for extra display devices,and has been widely applied in fields of art,design,advertisement,etc.The choices for target projection object for most projection mapping systems are static objects or dynamic rigid objects such as buildings or sculptures.In recent years,many researchers successfully performed projection mapping on dynamic human contour or face.Projection mapping on full human body will be a more prospective direction,as it could bring novel AR experience like real-time cloth switching on stage.However,projection mapping on full human body requires tracking high-dimensional deformable 3D human surface in real-time,which is hard to achieve with current algorithms and techniques.In this paper,we design and implement a Kinect based dynamic full human body projection mapping system,that given reference 3D human model,it could perform motion prediction,tracking and projection on moving human body in real scene with Kinect.To address the problem of real-time human tracking,we propose a deep neural network based fast matching algorithm,which first use a deep neural network to extract dense accurate human body feature descriptors for input human depth map and human model,and then perform fast hierarchical feature matching to find correspondences.The extracted feature descriptors are also used by human motion prediction in our system to estimated motion field from successive human depth maps,the motion field is then used to predict future human depth maps to reduce system delay.We verify the accuracy and efficiency of our human matching algorithm,and the effectiveness of our human motion prediction algorithm by experiments.We also demonstrate our system in real scenario.The main work and contributions of this paper are as follows:1)We design and implement a dynamic full human body projection mapping system,that given reference 3D human model,it could perform motion prediction,tracking,projection on human body in real-time with Kinect.We ensure the robustness and efficiency of our system by integrating and optimizing various algorithm in preprocessing,motion prediction,human tracking,and projection steps.2)We propose a deep neural network based fast human body matching algorithm.Trained by multiple human body segmentation tasks,a deep network could extract identical features for input human depth map and human model.The human body correspondences are then computed by fast hierarchical feature matching between the extracted features in real-time.3)We test our system with synthetic and real data.The results prove that our system can perform motion prediction,tracking,and projection on human in real-time.The matching error of our algorithm is acceptable,the motion prediction algorithm reduces projection delay.In conclusion,projection mapping on full human body,and its core human body matching algorithm could bring novel experiences to augmented reality.Our work of system design and implementation are of great research and engineer value.