Joint Multimedia Information Retrieval With Feature Learning

Multimedia information retrieval is typically modeled as discovering the latent information in multimedia data and achieving effective and efficient retrieval. Mul-timedia information retrieval generally consists of the feature learning aspect and the ranking model construction aspect.In principle, the goal of feature learning is to exploit the intrinsic structure property of multimedia data to obtain an effective-feature representation, while ranking model aims to encode the underlying ranking structure information of the data samples. To discover the features that are able to fit for the retrieval application, we establish a new feature learning model as a bet-ter data preprocessing approach. Besides, the interaction structure exists between the feature learning model and the ranking model:the obtained model from fea-ture learning is capable of reflecting the intrinsic ranking relationships among data samples; the ranking model construction is simultaneously able to provide extra regularizations, and thus leads to the feature representations with semantic mean-ings. Therefore, the feature learning model and the ranking model are mutually reinforced in theory. Based on the above observations, this thesis aims to propose a joint learning model, which obtains the ranking-specific features to further improve the retrieval performance as well as to increase the retrieval speed without losing the retrieval accuracy. Consequently, this thesis studies the multimedia information retrieval problem and has the following contributions:First, single-modal feature learning is exploited to construct effective feature representations. This thesis introduces a fine-grained grid-based blockwise alternat- ing least square approach, which solves a set of conditionally independent optimiza-tion subproblems instead of the original nonnegative matrix factorization problem for computing the subproblems in parallel.To further consider the consistent information of different modalities in the multimedia feature learning problem, the proposed approach maps the multi-modal features onto a consistent subspace to capture the generalities and differences of all the modalities. Meanwhile, this algorithm exploits a binary constraint and is parallelized in the distributed system, and thus greatly enhances the algorithm efficiency.After introducing the above feature learning approaches, this thesis focuses on designing joint ranking approaches with feature learning. First, this thesis proposes a hierarchical joint learning-to-rank with deep feature learning, which simultane-ously obtains a set of deep linear features and constructs structure-aware ranking models. Through a joint learning mechanism, the feature learning module and the ranking module are mutually reinforced, and meanwhile their underlying interaction relationships are reflected by solving an alternating optimization problem.At the same time, this thesis introduces a deep neural network from the feature group discovery point to adaptively explore the feature groups and their correspond-ing group-specific weights for relative importance evaluation. The obtained group information can be considered as latent variables by appending to the latent struc-tural SVM ranking module, and thus jointly models the group information discovery and learning-to-rank.