Video-based 3D Human Model Pose And Shape Reconstruction

Video-based 3D human model pose and shape reconstruction,which means extracting effective motion and shape information from the video of human motion,using the information to construct a three-dimensional human body model motion sequence with similar pose and shape to the video character,has important implications for 3D human modeling,model editing,and game production.Compared with the method of using special equipment or multi-view video images,the difficulties in the reconstruction of the pose and shape of the video-based 3D human body model are as follows: Firstly,the absence of depth information can result in the posture reversal.Secondly,it is easy for the limbs to penetrate each other which makes the shape of the reconstructed model inaccurate.Thirdly,it is difficult to handle the details of the head posture.Furthermore,the large amount of video data and the complex and changeable scene also makes the 3D human shapes processing a big challenge.This paper proposes a video-based 3D human pose and shape reconstruction method combining video coherence,automatic extraction of facial features and contours of video frames.Firstly,the coherence between frames in the video is used to constrain the differences between the corresponding 3D human model poses,so as to solve the problem of pose inversion caused by the lack of depth information.Secondly,we extract facial feature points of the person in the video frame,and then use these feature points to constrain the pose of the head in the three-dimensional human model,so as to adjust the head of the three-dimensional human model under the premise that the model pose is similar to the video character.Making the result of reconstruction more reasonable.Next,we use the two-dimensional joints extracted from the video frame to construct the suspected foreground and foreground regions and drive the GrabCut segmentation algorithm to automatically segment the regions of the human body in each frame,and extract the outlines.Through the iterative method,the extracted contour lines are used to constrain the shape of the three-dimensional human body model to adjust its shape and size so that its shape is more similar to the characters in the video frame.Finally,we implement a video-based 3D human pose and shape reconstruction system to verify the proposed reconstruction method.The system can automatically extract two-dimensional joint points,facial feature points,and two-dimensional contour lines from the input video.And it can reconstruct a three-dimensional human body model with similar posture and shape to the people in the video.Compared to other methods using wearable devices or multi-view images or video,our method adapts to various ordinary videos.In addition,we compare our method with two reconstruction methods.A large number of experimental results show that our method has high accuracy both in pose and shape reconstruction,and can accurately reconstruct the head pose of the model.What's more,our method can effectively avoid the problem of gesture rollover.