Research Of Image Retrieval Based On Sketch Semantics

In the research field of sketch based image retrieval(SBIR),where the sketches are treated as the queries to search for the natural photos in the natural image dataset,sketches show a highly abstract visual appearance versus natural photos,and fewer contexts can be extracted as descriptors using the existing methods.To improve the description power of features for sketches,we firstly proposed a conception of “multi-layers” semantic feature of sketches,and based on the semantics feature of sketches,we built a sketch based multi-layer fusion convolutional neural network to learn and extract the deep features from different layers of sketches.As a result,compared with the international recognized algorithms,our algorithm improves 30%~40% versus hand-crafted features.Another question is that: for the same object,different people show a huge gap among their drawing skills and recognitions,so the sketches from them are hard to uniform and standardize,not to mention using them to search the colorful photos.On this point,we designed a new strategy of mapping the sketches and photos(color images)to the same visual filed on different layers,that is,we implemented the separate operation on both sketches and photos.Finally,our strategy realized a robust and precise retrieval performance between sketches and photos,which showed 7.5%~30% improvements versus deep features on the retrieval accuracy.Then,based on the above strategy and thoughts on cross modeling between sketches and photos,we designed a new framework called deep multi-layer fusion convolutional neural network.Then,in order to acquire the unique and stable feature representation for the multi-layer visual representations,we presented a new feature fusion method to condense and concatenate features from the different layers by PCA algorithm.Experiments show that our feature fusion method realizes the best cross retrieval results among the current feature fusion methods.Compared with existing approaches,our “multi-layer” semantic feature can catch and extract the important spatial and semantic features from sketches;Then,the feature fusion strategy in our paper is not only able to simplify the complexity of multi-layer deep features,but also can improve the efficiency and accuracy of feature matching;What's more,our multi-layer mapping strategy between sketches and photos provides a stable and high-performance blueprint for research field of the sketch based image retrieval.