Research On Multiview3D Human Motion Capture

For several decades, vision-based Human Motion Capture (HMC) is a hot issue in computer vision area. But, after years of intensive investigation, it is still an open question due to the presence of cluttered background, ambiguities from3D-2D projection, occlusion and self-occlusion, high-dimensional state space, etc.In this thesis, we focus on three important issues of vision-based HMC that are how to extract reliable human body silhouettes under complex scenes, how to eliminate the assumption about video synchronization in multi-view HMC, and how to improve the robustness in human pose estimation, and aim to reduce the limitation of vision-based HMC system in real-world application and provide a HMC solution which is capable of providing robust and accurate HMC data with more flexibility. The main contributions of the thesis are summarized as follows:1. Propose a new foreground segmentation method based on Bilayer Gaussian Mixture Model (BGMM) and Markov Random Field (MRF), i.e. BGMM-MRF, for dynamic background. It can not only handle the repetitive movements in dynamic scenes, but also the sudden or gradual illumination changes. Compared to the existing methods that only consider either the repetitive movements and gradual changes or sudden illumination changes, it can handle more complex scenes. Benefiting from its GMM and MRF basis, it also fully exploits both temporal and spatial relations in pixels, which can be used to correct the false detection and holes in foreground objects, and provide coherent and accurate segmentation results.2. Propose a new foreground segmentation method based on Hierarchical MRF (HMRF). In this work, a HMRF statistical model is developed to infer both the background state within each image region and the segmentation of the image simultaneously. Compared to BGMM-MRF, by estimating background state in region-level rather than frame-level, it allows different regions in image under different background states. Thus, it can handle more different background states and provide more robust and accurate results.3. Present a HMC method based on unsynchronized videos. Using the sudden illumination changes produced while recording human motion videos, it estimates the temporal alignment between different videos. Based on the estimated temporal alignments, a human pose estimation method that is robust to illumination changes is proposed to obtain human motion data from multi-view videos under sudden illumination changes. In this way, the assumption about video synchronization in HMC system is removed. This eliminates the requirement of expensive synchronization hardware and improves the mobile flexibility of the whole system.4. Propose a new human pose estimation method based on Dynamic MRF with Color Constraints (CC-DMRF). By introducing color constraints into pose estimation, it can solve the ambiguity within the matching between human body model and voxel data when different human body parts making contacts, and improve the robustness of human pose estimation.5. Develop a multi-view markerless HMC system which integrates all the algorithms proposed in the thesis into a unified platform. By applying the system into protection of intangible cultural heritage and animation production, we verify practicability of the system.