The information geometry of EM variants for speech and image processing

Estimation from incomplete data arises in many statistical applications such as speech recognition and Positron Emission Tomography (PET). We analyze the Expectation Maximization (EM) algorithm for solving this problem using the information geometry of Csiszár and Tusnády, in order to understand how it may be extended. Under this framework, the EM algorithm is viewed as the alternating minimization (forward and backward projection) under the Kullback-Leibler divergence between a family of models and another family defined by the observed data. The GEM variant of the EM algorithm corresponds to replacing the backward projection with a step that reduces the divergence rather than minimizing it.; The convergence properties of the EM algorithm are retained when the forward projection is similarly extended. The incremental EM algorithm of Neal and Hinton results from such an extension. Applying this algorithm to PET yields a block-iterative algorithm similar to Ordered Subsets EM (OS-EM), Block Iterative EMML (BI-EMML), and the Row Action ML Algorithm (RAMLA). Unlike these algorithms, incremental EM is guaranteed to give maximum likelihood solutions in all cases.; Extending the family defined by the data yields robust procedures for estimation from small amounts of data. Applying such a procedure for rapid adaptation of hidden Markov models in speech recognition yields a robust adaptation scheme termed Discounted Likelihood Linear Regression (DLLR), which is a variant of the popular Maximum Likelihood Linear Regression (MLLR) scheme.; EM variants based on extensions of the information divergence such as the f-divergences is also examined. In particular, we derive a family of EM variants that provide a continuous family of estimators that range from the maximum likelihood estimator to the chi-squared estimator, which extends some existing estimators to incomplete data problems.