The Research Of Structures For Latent Variables In Generative Models

Generative models are important methods in Machine Leaning,it is used to model data or acts as a mid step in conditional probability density functions.Generative models can indicate the joint probability distribution between observations and target values,and its conditional distribution can be defined by the Bayesian rule.This model supposes that the whole dataset is generated from one probability model,this assumption links the unlabeled data with the learning target.The typical generative models include Gaussian Mixture Model(GMM),Variational Autoencoders(VAEs)and Latent Dirichlet Allocation(LDA).In the process of constructing generative models for different dataset,the prior of latent variables can significantly influences the final generative likelihood performance.Mixture of models is a way to improve the flexibility and likelihood of generative models.In this paper,researchers improved the prior model of latent variables in GMM and VAEs,furthermore we proposed a latent Gaussian-Multinomial Generative model(LGMG).The details of these works as below.Firstly,in order to improve the estimation of hidden number for GMM,this paper proposed the splitting and merging algorithm based on KS test for GMM(KSGMM).The KSGMM was developed on the base of Expectation Maximization Algorithm(EM),and it took the Minimum Description Length(MDL)as its objective function to balance the data fitness and model complexity.In KSGMM,the information entropy and KS test were used as criteria to identify the error Gaussian models,which improved the estimation precision of latent variables.Secondly,this paper proposed a mixture variational autoencoders(MVAEs).MVAEs assumed that the dataset is generated from a mixture of models,and it persisted the continue latent variable as representation of samples and introduced a discrete variable as indicator for mixture of models.MVAEs used full-connected neural networks to estimate models of latent variables and the samples.In the stochastic gradient variational Bayes,the reparameterization trick and Mento Carlo sampling methods were combined.In experiments,the MNIST and OMNIGLOT datasets were used to analyse the performance of MVAEs,CVAEs,VAEs,SB-VAE,VAE_IAF and GMAVEs.Thirdly,based on the works in MVAEs this paper also proposed a novel model called latent Gaussian-Multinomial Generative model(LGMG).LGMG is a three-layers Bayesian model used for modeling on image semantic annotations.In comparison with trational LDA-based models and deep learning models,LGMG needn't to segment the image to multiple instances instead of summarizing the abstract semantic information by a Gaussian latent variable.This paper compared the annotated results of LGMG with tr-mmLDA,cLDA and the likelihood performance with VAEs,cVAEs.