| Prosody (the melody and rhythm of natural speech), although important for human speech recognition, has not been fully utilized in large vocabulary continuous speech recognition. In this dissertation, we propose a novel "prosody-dependent speech recognition" framework, in which word and prosody are recognized simultaneously for the purpose of improving word recognition accuracy. We review the linguistic literature on how prosody is used in human speech communication, what the structure and function of prosody is, and how prosody affects acoustic realization of the segmental and suprasegmental units of speech. We conduct information-theoretic analysis, proving that when prosody is modeled, better word recognition can be achieved through the interaction between the acoustic model and the language model. We conduct detailed experiments to determine the set of allophonic HMMs or probability distributions that are sensitive to prosody, under the guidance of linguistic prior knowledge and empirical selection rules. We measure the effects of prosody on the language modeling and on the pronunciation modeling, and propose a factored approach, which leverages the strong dependence of prosody over syntax, to improve the robustness of the N-gram language modeling. We also develop an automatic prosody labeling system as a way to reduce the human labeling cost based on a simplified version of the Tones and Break Indices system. In the word recognition experiment on the Boston University Radio News Corpus, we find that our system is able to reduce word error rate by as much as an absolute 11%, as compared with a conventional prosody-independent HMM-based automatic speech recognizer that has comparable parameter count. Our research confirms that explicit modeling of prosody in HMM based automatic speech recognizers can improve word recognition on American Radio News speech. |
