Design of pattern recognition systems using deterministic annealing: Applications in speech recognition, regression and data compression

This work investigates the application of the deterministic annealing (DA) optimization approach to problems in speech recognition, regression, and data compression. DA, which is based on the principle of randomization and grounded in fundamental concepts from information theory and statistical physics was first proposed for data clustering and vector quantization and recently extended to solve pattern recognition problems such as the design of structured classifiers. Here we significantly extend the scope of DA to solve the problems of designing regression functions and hidden Markov model (HMM) classifiers used in speech recognition applications. In the context of source coding, we propose a DA approach to attack the difficult design problem for the novel generalized vector quantizer (GVQ) structure.;In classifier and regression function design problems, the true cost function, which is an average of a recognition cost over the training set, is difficult to optimize directly because of the complex nature of the cost surface. Standard design methods usually abandon the true cost in favor of other design objectives, which although inaccurate, are easier to optimize. In HMM classifier design, for example, the true minimum classification error goal is usually replaced by the suboptimal maximum likelihood objective. The choice of this inaccurate objective leads to suboptimal performance both inside and outside the training set.;With the discovery of the powerful DA optimization approach however, one can minimize the true recognition cost directly. In addition, the DA formulation for classifier and regression function design embeds the search for small models within the optimization problem. The choice of small models leads to better generalization outside the training set. We consider regression functions that are based on the piece-wise and the mixture of experts models and demonstrate the advantages of the DA design approach.;We also present here, an important extension of the DA method to solve the difficult problem of designing HMM-based speech recognition systems. The DA method, unlike many other HMM design methods, attacks the classifier's error rate directly resulting in improvements by orders of 2--3 in error rate, relative to competing design methods. The novel HMM design algorithm can be implemented by a low complexity forward-backward procedure similar to the Baum-Welch re-estimation used by the standard but suboptimal maximum likelihood design method.;In the context of data compression, we apply DA to solve the problem of designing generalized vector quantizers (GVQ). GVQ deals with the quantization of a random data vector, Y given a statistically related vector, X. GVQ, which includes the regular vector quantizer (VQ) as a de-generate special case (corresponding to X = Y), is a powerful paradigm which applies to many applications including the quantization of noisy sources and nonlinear prediction.