Recovering Structure And Motion Of 3D Non-rigid Object From Image Sequences

One of the study object of computer vision is recovering both structure and motion from 2D image sequences. During the last two decades, many approaches have been proposed for different applications. Among them, factorization based methods are widely studied and attracted much attentions due to their good robustness and accuracy.The traditional methods of 3D reconstruction work mainly for static rigid object. While in real world, most objects and scenes are non-rigid and dynamic. Many previous methods on this problem are hardly applied to reconstruction of this object. How to recover both structure and motion of non-rigid object from image sequences is a study hotspot of computer vision and pattern recognition.There exists two bottlenecks in reconstruction of recovering structure and motion of 3D non-rigid object:one is the choice of model, the other is system robustness. Previous reconstruction methods are all based on affine camera model. This is a zero-order(weak perspective)or first-order(paraperspective)approximation to the general perspective projection model and is only valid when the depth variation of the object is small compared to the distance between the object and the camera. Usually, it does not satisfy the assumption for many image sequences in real life. Therefore, there exists large reconstruction errors for the result, especially the depth variation of the object is large or the distance between the object and the camera is small. Otherwise, the methods of 3D reconstruction for non-rigid object all assume that the match of tracked feature points is known and these features are visible in all images. However, match mistakes or missing of features in some frames are often appear due to occlusions or tracking methods failure.In this case, the existent methods can't be used directly.Aimed at above problems, the paper addresses the problem of recovering structure and motion of 3D non-rigid object from uncalibrated image sequences under perspective projection model. In addition, the paper also studys robustness reconstruction methods to solve the problem of tracking error of features and missing data.1. The paper proposes a recursive algorithm to estimate 3D structure and motion of non-rigid object from a monocular video sequences and update previous non-rigid factorization methods from weak perspective assumption to the case of perspective projection. Accordingly, the precision of the reconstruction result is improved.2. The paper proposes a quasi-perspective projection model to recovery structure and motion of rigid and non-rigid objects. First, under the assumption that the camera is far away from the object with small rotations, we propose and prove that the imaging process can be modeled by quasi-perspective projection. The model is more accurate than affine camera model since the projective depths are implicitly embedded in the shape matrix. However, it is computationally as cheap as affine. Second, we apply the model to the factorization algorithm and establish the framework of rigid and non-rigid factorization under quasi-perspective assumption. Third, we propose a new and robust method to recover the transformation matrix that upgrades the factorization to the Euclidean space.3. The paper proposes a constrained power factorization algorithm that combines the orthonormal constraint and the replicated block structure of the motion matrix directly into the iterations. The proposed algorithm overcomes the limitations of previous SVD based methods. It is easy to implement and can even cope with the tracking matrix with missing data. Based on the solutions of the CPF, a novel sequential factorization technique is proposed to recover the shape and motion of new frames in realtime.4. The paper proposes two new algorithms to improve the performance of perspective factorization. First, we propose to initialize the projective depths via a projective structure reconstructed from two views with large camera movement, then optimize the depths iteratively by minimizing reprojection residues. The algorithm is more accurate and converges quickly. Second, we propose a self-calibration method based on Kruppa constraints to deal with more general camera model. The Euclidean structure is then recovered from factorization of the normalized tracking matrix.Extensive experiments on synthetic data and real sequences validate the effectiveness of the proposed algorithm and show noticeable improvements over the previous methods and provide reference to the further study and application.