Research On Key Techniques Of Visual Scene Understanding And Interaction

With the rapid development of existing multimedia devices and related technolo-gies,there is an increasing demand in visual scene understanding and interacting with the 3D reconstructed scene.Nowadays,technologies of visual scene understanding and interaction are applied in many fields,such as art design,robotics,industrial manufac-turing and virtual reality.However,the current framework is still far away from users'expectation in accuracy,efficiency and generality.Therefore,we focus on some major issues to seek solutions,such as precisely understanding the scene,efficiently interac-tion with the reconstructed scene and construction of the framework to achieve scene understanding and interaction.In this thesis,we aim to construct such framework and research on several related techniques,such as object detection,thin or restoration of scene characters,3D mesh interactive editing.Additionally,we design a platform to assist sculpture design involving technologies in scene understanding and interaction.Regarding these issues,this thesis engages in the following work:(1)We propose a novel approach for multiclass object detection by combining lo-cal appearances and contextual constraints.We first construct a multiclass Hough for-est of local patches,which can well deal with multiclass object deformations and local appearance variations.Then,we propose a context model to capture relative location constraints.Finally,multiclass objects are detected with a greedy search algorithm to efficiently combine local appearances and contextual constraints.(2)We propose to explore ring radius transform(RRT)to obtain the thinning and reconstruction results of scene characters.Thinning that preserves visual topolo-gy of characters is important for many document analysis and computer vision tasks.We firstly propose to identify medial axis(middle pixels between stroke contours)for characters with arbitrary orientations based on radius map generated by RRT.Next,we use iterative-maximal-growing strategy to restore missing medial axis pixels based on its neighborhood.After eliminating false segments based on clustering,we reconstruct the shape of the character through radius values of medial axis pixels.(3)We propose to directly restore character contours in video/scene images from gray domains.Restored characters as input could significantly improve results of cur-rent methods on text detection and recognition.We firstly utilize the strengths of zero crossing points computed by Laplacian Operator to identify stroke candidate pixels(SPC).Then,we propose symmetry features in gradient and Fourier domain to identify probable stroke candidate pair(PSCP).The similar symmetry is also adopted to choose seed stroke candidate pair(SSCP)from set of PSCP.Finally,an iterative algorithm is proposed for SSCP to restore character contours.(4)We propose a reduced and real-time mesh editing method,supporting incre-mental mesh editing where the user can freely define different handle sets to edit d-ifferent parts of the shape without waiting for long re-computation.We define the propagation fields based on spectrum of Laplacian matrix and adopt the transforma-tions of isolines sampled from the fields as the deformation descriptors.In this way,the deformation descriptors are pre-computed before handle specification.During in-teractive manipulation,constraints generated from the handles are incorporated into the deformation system in real time.(5)Based on the background of sculpture design,we propose a novel platform to assist sculpture pose design in realtime,which constructs a framework for scene understanding and interaction.The proposed platform allows sculptors to firstly recon-struct 3D sculpture model from scene images,and then freely edit poses of the recon-structed sculpture through vision interactions.We firstly initialize sculpture artwork by co-segmenting and incrementally reconstructing detail-abundant 3D sculpture models from scene images of different views.Next,we construct a Laplacian matrix of the reconstructed sculpture to automatically search for editing points.Finally,we propose a novel real-time Kinect-driven sculpture pose editing method,supporting sculptors to intuitively edit poses of sculptures through vision interactions.