Natural Image Stitching Based On Global Vanishing Point Constraints

In recent years,with the continuing development of Virtual Reality(VR)and Augmented Reality(AR),users have increasing demand for the multimedia immersive experience.Meanwhile,various intelligent algorithms represented by the deep learning technique develop from perspective image based methods to panoramic image based methods,which demands for a flexible and simple method to generate panoramic images with high quality.The technique of image stitching gains extensive attention recently since its advantages,such as high flexibility and simplicity of use.In this paper,we mainly focus on the problem of generating panoramas with natural appearances by image stitching.We propose to take the vanishing point(VP)as an effective global constraint,and develop a novel similarity prior estimation method for natural image stitching.The VP guidance is exploited by taking its two advantages:(1)the VP corresponding to a cluster of parallel lines often contains significant orientation clues,and can reflect the context information of the image scene,based on which we first propose to roughly estimate the image similarity prior with the VP guidance;(2)under the Manhattan assumption,VPs from different images are globally consistent.Starting from this property,we then achieve a robust estimation for image similarity prior.After that,the determined similarity prior is feed into a classical mesh deformation framework as a global similarity constraint to stitch multiple images into a panorama,which has high alignment accuracy in the image overlapping region and is naturally looking in the non-overlapping region.Moreover,in this paper we further study the proposed method in the aspect of robustness under the quasi-Manhattan scene and the reversibility under the non-Manhattan scene.Experimental results show that our method performs stably in both Manhattan and quasi-Manhattan scenes,and can effectively fall back to the standard mesh deformation scheme when the scene disobeys the Manhattan assumption.We compared our method with other existing methods on both synthetic and real images,including typical indoor and street-view scenes.Besides,in order to quantitatively evaluate the naturalness of produced panoramas,we propose two evaluation metrics LD and GDIC.They measure the local perspective distortion and the global direction consistency existing in the panorama,respectively.The quantitative and qualitative comparisons consistently demonstrate the superiority of our proposed method.Besides,we conduct experiments on wide-baseline images,and the experimental results show that our method can effectively cope with this challenging scene.Moreover,we perform a user study to further compare our method with different natural image stitching approaches.The result of user study demonstrates that our method is preferred by users.Also,our proposed two evaluation metrics are valid for panorama naturalness evaluation to some extent.