Information fusion for monolingual and cross-language spoken document retrieval

Spoken document retrieval (SDR) is an important technique that enables relevant information to be searched from spoken data archives. With the advent of Internet and multimedia technologies, the amount of available information sources in various media (text, audio, video etc.) is increasing very quickly. In order to make these available sources of information more readily usable, there are two major issues worth of investigations. First of all, since there is large quantity of information stored in audio format in addition to textual format, capability of searching among spoken data will allow one to make use of this large pool of data resources. Furthermore, the available information can also be presented in different languages. To handle this problem, information searching across different languages is also important.; In an information retrieval system, information is usually extracted by matching the indexing units at different scales. These scales include words as well as subwords (e.g. phonemes, characters or syllables). Among these indexing units, words are known to achieve higher precision in retrieval whilst subwords are more robust to errors within documents (e.g. recognition errors). In order to take advantage of these indexing units at multiple scales, we propose to apply information fusion to these units by multi-scale retrieval. Multi-scale retrieval refers to the use of both word and subword units for retrieval.; In this thesis, we shall present thorough investigations for both monolingual and cross-language spoken document retrieval tasks using the proposed multi-scale approach. We have investigated the multi-scale retrieval performances of the two most commonly used retrieval models, vector space model (VSM) and HMM-based model. The experiments are carried out using both monolingual Chinese (both Cantonese and Mandarin) and cross-language (English searching Mandarin) data archives. Specifically, in order to achieve multi-scale SDR with the HMM-based model, we have taken advantage of the probabilistic nature of this retrieval model to extend it for both cross-language information retrieval and subword scale retrieval. Furthermore, the success of multi-scale retrieval encourages us to extend this approach to multi-model as well as multi-scale and multi-model retrieval for SDR. Experiments on both monolingual and cross-language SDR tasks have demonstrated that the benefits of using these information fusion approaches are remarkable.