Variational Inference To Supervised Dirichlet Process Mixtures Of Principle Component Analysers

Compared with traditional finite mixture model, dirichlet process mixture(DPM) model have the advantages that can solve the problem of the unknown number of clusters, and allow the model complexity to grow as more data are observed.As a result, it has been widely used in recent years. By combining DPM and supervised learning model, supervised dirichlet process mixture(SDPM) model can model the covariates and response variable nonparametrically using DPM, and learn a local expert model within each component. The linear model will become globally nonlinear if the mixture has more than one component, which extends the learning ability and flexibility of the model. However, SDPM would suffer when facing high-dimensional data, since it regards the original covariates as input space.In order to solve this problem, we introduce probablistic principle component analysis(PPCA) into SDPM and propose a new model called SDPM-PCA. As a commonly used dimension reduction algorithm, PPCA can project high-dimensional data to lower dimension, which would accelerate the training speed and avoid overfitting. SDPM-PCA assumes the covariates and response variable are generated sperately throught the latent variable of PPCA, and nonparametrically modeled using the dirichlet process mixture. By jointly learn the latent variable, cluster label and response variable, SDPM-PCA performs locally dimension reduction within each mixture component, and learns a supervised model based on the latent variable. In this way, SDPM-PCA improves the performance of both dimension reduction and prediction on high dimension data with all advantages of SDPM. We also develop an inference algorithm for SDPM-PCA based on variation inference. It provides faster training speed and deterministic approximation compared to sampling algorithms based on MCMC method.Finally, we instance SDPM-PCA in regression problem with a bayesian linear regression model. We test it on several real world datasets and compare the prediction performance with SDPM and other regular regression model. Experiment results show that by setting properly latent dimension number, SDPM-PCA would provide better dimension reduction performance and usually have better prediction performance on high dimension regression problem.