Researches On Linear, Kernel Gaussian Models And Their Mixtures

A powerful approach to probabilistic modeling involves supplementing a set of observed variables with additional latent, or hidden, variables. By defining a joint distribution over visible and latent variables, the corresponding distribution of the observed variables is then obtained by marginalization. The well-known model is the so-called Latent Variable Models (LVM). In this paper, we will discuss a special LVM with Gaussian distribution from the following three perspectives. Firstly, the relation of the existing linear Gaussian models is demonstrated, and the equivalence between Probabilistic Principal Component Analysis (PPCA) and Probabilistic Subspace Analysis (PSA) is proved. We generalize the existing mixture of linear Gaussian models (MLGM) by introducing some new assumptions of noise models. Conclusively a common framework is built to unify the mixture models.Secondly, the new nonlinear latent variable model, called Kernel Gaussian Models (KGM), is proposed. The KGM has two characters: (1)In KGM, the observed variable is mapped into a high-dimensional space via nonlinear functions, while the latent variable is processed by nonlinear functions in traditional nonlinear approaches; (2)It can be efficiently estimated by introducing the well-known kernel trick. Moreover, the mixture of kernel Gaussian models (MKGM) is proposed to realize nonlinear probability density estimation. Compared with MLGM, MKGM can obtain better performance when dealing with nonlinear problems.Finally, in theory, the well-known Spectral Clustering is proved to be a special case of Weighted Kernel Principal Component Analysis, which is one kind of KGM. From this perspective of view, one speedup algorithm of Spectral Clustering is proposed. In this paper, many experiments, such as clustering, image compression, image segmentation and pattern classification, demonstrate the efficiency and accuracy of the novel models and the new interpretation appearing in this paper.