Research On Key Techniques Of Object In-depth Retrieval From Images

With the proliferation of image capture devices and mobile internet, we can see the explosive growth of the number of images. Retrieving object efficiently and precisely from such huge number of images have attracted lots of interest in recent years because of its academic value and wide applications. And with the ever-growing requirements of users, the concept of object in-depth retrieval is coming into peoples’eyes. Generally, object in-depth retrieval system can be defined in two aspects:(1) it should be able to generate finer image annotation. Finer image annotation is consists of pixel-level annotation of object area (segmentation) and object part annotation. These finer annotation allows retrieval system returns finer results; (2) it should be able to understand finer user input. For instance, a sketch, which reflects the object’s detail shape, pose and view angle, can be considered as an input. The ability of understanding finer input allows retrieval system returns results that highly match the user input. In generally, compared with traditional retrieval system, object in-depth retrieval system can return the results that more qualified with users’requirements, and largely improve retrieval efficiency by avoiding re-screening the returned results. Therefore, our work focus on object in-depth retrieval and the following results have been achieved:(1) We propose an annotation framework that based on superpixel and improved AND/OR graph for the problem of pixel-level annotation and object diversity. The diversity of object appearance and pose largely decreases the performance and increase the difficulty of pixel-level annotation. To solve this problem, we divide the object into a set of object parts, and propose an improved AND/OR graph to organize the relationships between object parts. A corresponding rapid inference algorithm is introduced to find the optimal object parts. Considering the requirement of pixel-level annotation, the candidate object part set is generated based on the shape matching score between superpixels and pre-defined part templates. By integrating superpixle and improved AND/OR graph model, our method can produce pixel-level annotation, and is robust to object diversity. The experimental results on three public datasets indicate that our method is able to annotate both object location and object area, and being robust to object diversity.(2) We propose a "predict-and-enhance" method to improve the robustness of object part annotation. Compared with whole object, there is smaller shape variance of object part, which is helpful for object part annotation. However, object part annotation is more vulnerable to vague contours and noisy edges due to fewer discriminative features. Based on the analysis above, we aim at improving the robustness of object part annotation by enhancing the object part contours. An edge pattern learning algorithm is proposed to automatically learn edge patterns from positive training set. Based on the learned edge patterns, we develop a "predicting-and-enhancing" method to predict the most possible object part contours for each input image, and then enhance the predicted contours and suppress the noisy edges. By doing this, the robustness of object part annotation system has been improved. The experimental results on INRIA and TUD datasets validate the effectiveness of our method.(3) We propose a boundary selection algorithm to solve the noisy edge problem in sketch-based object retrieval. There is large appearance gap between sketches and natural images because natural images contain many noisy edges. How to alleviate the impact of noisy edges is the key to improve the retrieval performance. We treat the sketches and extracted edges of natural images as a set of line segments, and develop a new HLR (histogram of line relationship) descriptor to describe object shape by capturing the relationship between line segments. Noticing that the extracted edges could contain noisy edges from object detail and background, based on the HLR descriptor, we propose an object boundary selection algorithm to make the retrieval system focus on object boundaries and ignore noisy edges. Hypotheses for each HLR descriptor is generated, each of which corresponds to a result of boundary selection. The selection problem is therefore can be formulated as finding the best combination of hypotheses, and a fast method is introduced to solve this optimization problem. The experimental results indicate both the HLR descriptor and boundary selection algorithm improve the retrieval performance, the whole framework significantly enhances the robustness of retrieval system to noisy edges.(4) We propose an optimal partial matching algorithm to solve the edge fallibility problem of sketch-based image retrieval. Edge extraction from natural images not only bring noisy edges, but also result in contour edge missing, which is referred as edge fallibility. Edge fallibility enlarge the appearance gap between sketches and natural image and degrade the retrieval performance. Noisy edges make the extracted edges become a superset of sketches, and contour missing makes the extracted edges become a subset of sketches. The matching task between sketches and extracted edges is therefore can be formulated as an optimal partial matching problem. We propose a new SP (structure point) descriptor and hierarchal matching algorithm to achieve partial matching. SP descriptor is designed to capture the local structure of the object through describing the intersections of line segments. The hierarchal matching algorithm decompose a SP into a hierarchal descriptor set, and a top-down hierarchal matching algorithm is applied to achieve the partial matching between SPs. The experimental results on three datasets validate our framework.