Recognition, mining, synthesis and estimation (RMSE) for large-scale visual data using multilinear models

In this dissertation, we show the power of multilinear models in recognition, mining, synthesis, and estimation (RMSE) for large-scale tensor visual data in computer vision and graphics. We develop novel algorithms based on multilinear models to efficiently explore the relationship of multiple factors within the data and investigate some challenging applications in object recognition, video mining, image-based rendering, and 3D scene modeling. In object recognition, we present a general framework for dimensionality reduction using Datum-as-Is representation. Our approach works directly on the multi-dimensional form of the data (matrix in 2D and tensor in higher dimensions). This helps exploit spatio-temporal redundancies with less information loss than traditional image-as-vector methods. In video mining, we propose a novel incremental subspace learning algorithm to handle large scale or streaming tensor data. The algorithm updates the subspace matrices with sequential loading of data to reduce computational and storage complexities. We apply our algorithm to human action categorization and efficient video words retrieval for texture synthesis. In image-based rendering, we describe a novel out-of-core algorithm for higher-order tensor approximation. Our approach can handle very large dataset, synthesize novel images efficiently, and control each mode of the high-order data to obtain realistic rendering effects. In 3D scene modeling, we model the observed surface radiance tensor as the tensor product of a light transport tensor, and illumination and texture matrices, such that we can simultaneously estimate the illumination and texture from given multiview images captured under a single illumination setting.