Automatic Annotation Of Image Semantics Based On Semantic Context Modeling

Automatic Image Annotation (AIA) has attracted increasing attentions in recent years due to its potential in many interesting applications, such as keyword based image and video retrieval and browsing. However, a major bottleneck of AIA is the so-called semantic gap problem between visual perception and high-level semantics. To deal with this challenge, various AIA methods, mostly based on generative models and discriminative models, have been proposed in the current literature. Besides, the relationships between semantic concepts have been utilized in AIA and bring promising results. Semantic context modeling has been integrated with both generative models and discriminative models to leverage the learning power of AIA methods.This thesis presents a novel Markov Random Field (MRF) annotation framework for semantic context modeling in AIA. Different for the previous MRFs in vision recognition which model the spatial relationships between image pixels or regions, our MRF is built over semantic concepts to model the interactions between them. Specifically, the sites in our MRF correspond to semantic concepts and the edges represent the correlations between concepts. A binary label is associated with each site to indicate the presence or absence of the corresponding concept in an image.We propose a novel Multiple Markov Random Field (MMRF) contextual model for semantic context modeling in our MRF annotation framework. MMRF builds semantic level MRFs to refine the annotation results of traditional generative models in AIA. Specifically, we propose new potential functions based on joint probabilities of image visual features and semantic concepts estimated by some generative model in AIA. We build one MRF for each semantic concept to capture different semantics among them. Besides, we efficiently solve the parameter estimation and model inference problems of MMRF.We propose a novel discriminative Conditional Random Field model for semantic context modeling in our MRF annotation framework, called Maximal Margin Conditional Random Field (MMCRF) contextual model. Our model captures the interactions between semantic concepts from both semantic level and visual level in an integrated manner. Specifically, the potential functions are designed based on linear discriminative models, which enable us to propose a novel decoupled Hinge loss function for maximal margin parameter estimation. We train the model by solving a set of independent quadratic programming problems with our derived contextual kernel. Extensive experiments have been conducted on commonly used benchmarks:Corel and TRECVID-2005 data sets to evaluate the annotation performance of MMRF and MMCRF. The experimental results show that compared with the state-of-the-art methods in AIA, our methods achieved significant improvement on annotation performance. Especially, MMRF achieved 0.36 and 0.31 in average recall arid average precision respectively on 263 keywords in Corel data set. This remains a very competitive performance on Corel data set.