Human Motion Tracking Based On Multi-view Images

In the field of computer vision,the perception of a three-dimensional scene is a hot issue,and people in the scene are generally the focus of attention.At present,the three-dimensional information of the human has broad research value and application prospects in clothing design,virtual fitting,film and television game production,etc.However,due to the lack of high-precision vision cameras,structured light cameras and other equipment,it is difficult to reconstruct a high-precision three-dimensional human model.In addition,high-precision three-dimensional reconstruction also requires expensive hardware equipment and a lot of calculation costs.Even so,the reconstructed three-dimensional model is difficult to perfectly express the three-dimensional information of the scene.Therefore,the free-view synthesis method is proposed for synthesizing novel view images.This method does not rely on high-quality three-dimensional models.And the synthetic free-view images are derived from real scene images,which can give people a more realistic visual experience.Based on the background mentioned above,this paper focuses on three-dimensional human modeling and free-view synthesis.First of all,we collect data based on multi-view camera synchronous acquisition system,and carry out a series of image processing operations to prepare for the next work.Then,the 3D model of human is reconstructed based on the parametric model SMPL to express the shape and posture of human,and the human model is updated to track the human motion in time.Finally,in order to overcome the defects of human 3D model and improve the visual effect of viewing,free-view synthesis method is used to generate human free-view video.The specific work is as follows:(1)SMPL based human 3D modeling and motion tracking.For 3D reconstruction of human,the human pose is very important.Therefore,2D human pose estimation firstly is performed on multi-view images,and 3D human pose is estimated by combining multi-view 2D human pose.Next,the statistical model SMPL is used to express the shape and pose of performers.Then,the shape and pose of human body are adjusted and optimized according to the sparse point cloud reconstructed from multi-view images.Finally,the surface of SMPL model is deformed to fit the real point cloud to express the three-dimensional information of human.(2)Multi-view based static human free-view image synthesis.For free-view synthesis task,we focus on the human region in the image.Firstly,the human portrait segmentation technology is applied to separate the effective image regions.And according to the camera parameters of different views,the nearest adjacent view to the target view is selected to synthesize the novel view image.Then,we project the reconstructed 3D human model to the adjacent view to get the depth map.According to the depth information,we map the human image from the adjacent view to the imaging plane of the novel view.Finally,the deep-blending network based on deep learning is applied to weight the image pixels from different views to synthesize the novel view image.(3)Multi-view based dynamic human free-view image synthesis.In the performer's motion video sequence,temporal information contribute to human free-view synthesis.However,we need to align the human images at different times.Therefore,we first use the pose of SMPL model to drive the human images to achieve preliminary alignment,and then use deformable convolution to achieve the feature alignment of human image.Finally,the Long Short-Term Memory network LSTM is used to combine the time consistency information to synthesize the novel view image.In this paper,we perform 3D human modeling based on SMPL model to describe the shape and posture of human in the process of motion.Finally,the free-view synthesis method is used to synthesize the image of the human at any time and any view,which allows the viewer to change the view arbitrarily in time and space,and provides a more real and convenient viewing experience.