Research On Semantic-based Cross-Media Consistency

With the rapid development of Internet technology and the popularity of digital capture devices, the presentation of information content is becoming more and more complex and diverse. An information entity can be described by combining text, image and video together. The existing single-media processing methods cannot effectively handle these complex entities. Therefore, new techniques have to be developed. Cross-media computing is just a good solution for handling heterogeneous information entities. As one of key components, cross-media consistency attempts to capture the consistent information among heterogeneous information entities. In the thesis, we fo-cus mainly on cross-media consistency problem. The main contributions are as follows:1. A novel text representation (TF-ICF) is proposed on basis of category semantic in-formation. Different from traditional method, we treat each category-specific dic-tionary as a document and make a statistic on all category-specific dictionaries. The experimental results show this method indeed improves the retrieval precision after introducing the semantic information with category.2. A novel semantic representation of cross-media, called accumulated reconstruction error vector (AREV), is proposed, which includes category-specific dictionary codebook learning, media sample reconstruction, and accumulative reconstruction error concatenation. Instead of directly learning the correlation relationship among heterogeneous items in the same semantic groups, the proposed method separately projects each heterogeneous space into the common shared feature space. Experi-ments on the commonly used datasets show the good performance in terms of ef-fectiveness and efficiency.3. We have also designed a logo detection and recognition algorithm based on cross-media consistency. The key idea is to reduce the scope of logo candidates by employing the semantic consistency among street-view images and their GPS loca-tion information. This algorithm contains three key components:mining the pro-posal regions, extracting CNN features of these proposal regions, detecting and lo-calizing the logo position by combining GPS information.