Semantic-aware data processing: Towards cross-modal multimedia analysis and content-based retrieval in distributed and mobile environments

In the past decade, distributed multimedia data processing, i.e. the management of multimedia data objects from distributed data sources, has experienced an explosive development because of the technological advances in distributed computing, network infrastructures, and multimedia streaming. With the proliferation of the third generation wireless networks, it is expected that multimedia data transmission and manipulation will achieve even larger growth in the next decade. The improved bandwidth and mobility of computing devices enable the possibility of accessing multimedia data "anytime, and anywhere", providing the foundation for mobile multimedia data management.;At the same time, due to the characteristics of multimedia data (e.g. large data volume and lack of accurate semantic content representation), the existing multimedia information retrieval systems cannot guarantee accuracy, efficiency, and robustness when performing distributed content-based retrieval. In addition, the mobile environment has put further requirements on the manipulation of multimedia data: (1) The mobile computing devices are often disconnected from the network for prolonged periods of time due to the battery power limitations; (2) The network infrastructure is organized in a peer-to-peer fashion, making the traditional centralized or flooding-based search schemes ineffective; (3) The mobile computing devices in the network have frequent relocations, making it important to consider location information in the query processing; (4) The limitation of system resources (e.g. bandwidth and storage) require efficient approaches for handling voluminous multimedia data; and (5) The information retrieval system needs to integrate the heterogeneous and autonomous mobile data sources to provide a global framework for content-based multimedia data access. Generally, the overall performance of multimedia information retrieval is greatly influenced by the emerging issues in mobile networks.;Despite the fact that a great deal of research has been done on multimedia data access, there has been little work done in integrating content-based multimedia retrieval in the mobile environment, especially in the wireless ad hoc networks. In addition, there is not much research work reported on the semantic analysis and content representation of mobile multimedia data. These research issues, however, are crucial for the successful and efficient multimedia communications in mobile networks. Therefore, it is highly necessary to investigate these challenges and devise a novel methodology for mobile multimedia data management.;This dissertation is intended to present and analyze a new semantic-aware multimedia representation and accessing model in distributed and mobile database environments. Semantic classification and categorization of the multimedia databases are based on the Summary Schemas Model (SSM). The ability of summarizing general information provides a promising mechanism to represent and access multimedia data entities. In this dissertation, we propose a logic-based model that can be integrated in the SSM and used as the paradigm for multimedia data content representation. This dissertation also investigates the feasibility of the proposed model, compares and contrasts it against several models as advanced in the literature.;To provide an efficient platform for multimedia information retrieval in ad hoc networks, we propose to cluster ad hoc multimedia databases based on their semantic contents, and construct a virtual hierarchical indexing infrastructure overlaid on the mobile databases. This clustering scheme uses a semantic-aware framework as the theoretical foundation for multimedia data organization. Several novel techniques are presented to facilitate the representation and manipulation of multimedia data in ad hoc networks: (1) using concise distribution expressions to represent the semantic similarity of multimedia data, (2) constructing clusters based on the semantic relationships between multimedia entities, (3) reducing the cost of content-based multimedia retrieval through the restriction of semantic distances, and (4) employing a self-adaptive mechanism that dynamically adjusts to the content and topology changes of the ad hoc networks.;As an extension to the aforementioned multimedia content representation model, we also presented a semantic-aware image caching scheme to facilitate content-based multimedia information retrieval in ad hoc networks. The caching scheme can efficiently utilize the cache space and significantly reduce the cost of image retrieval. It is based on several innovative ideas: (1) multi-level partitioning of the semantic space, (2) association and Bayesian probability based content prediction, (3) constraint-based representation method showing the semantic similarity between images, and (4) adaptive QoS-aware cache consistency maintenance.;Overall, the focus of this dissertation is to provide a semantic-aware framework that is capable of representing, organizing, and searching multimedia data objects in the distributed and mobile environments. The proposed framework is scalable, fault-tolerant, and efficient in performing content-based multimedia retrieval as demonstrated in our combination of theoretical analysis and extensive experimental studies.