Variational Auto-encoders Based On Gaussian Mixture Model

Unsupervised learning,as a way to learn the real world from unlabeled data,can liberate humans from tagging of data.Feynman said: What I cannot create,I do not understand.There are many ways to evaluate the quality of unsupervised learning,among which the generation task is the most direct one.Only when we can generate/create our real world,can we understand it completely.Therefore,generation model has become one of the most popular algorithms in unsupervised learning algorithm in recent years.this paper will introduce one of the most popular generation models in unsupervised learning of complicated distributions,namely variational auto-encoder,a model that can automatically generate data.It is to transform the complex distribution of high dimensional into a simple distribution of low dimension.Then the original image can be automatically generated from the simple distribution of low dimensional.At present,the posterior distribution for the hidden variable z mostly satisfies the simple single distribution,such as Gaussian distribution,which causes the result that the representation of low dimensional is too simple to satisfy well the original distribution of the latent variable.However,there are many non-Gauss morphological distributions in the real world.And in particular,for some highly skewed multimodal distributions,a Gauss approximation is often not sufficient.The hidden space of data sets can also be arbitrary and complex distribution.For this reason,in order to improve the flexibility of the posterior distribution,we changed the approximate posterior distribution to Gaussian mixture model instead of the primary single Gaussian distribution.The addition of the Gaussian mixture model greatly increased the lower bound of marginal log-likelihood measured in nats of the digits in the test datasets.In order to further improve the flexibility of the posterior distribution,we introduced a new method,namely Normalizing Flows,on the basis of Gaussian mixture model.Normalizing Flows can be used to specify any complex,flexible,scalable approximate posterior distribution,that is,a simple initialization density function can be transferred to a desired distribution by using a series of reversible transformations.In this paper,we derive the variational lower bound from the Variational auto-encoder based on Gaussian mixture model and obtain its corresponding optimization algorithm.As a result of the addition of Normalizing Flows,each single Gaussian in the Gaussian mixture model can approximate the full-covariance matrix,that is,the covariance matrix of all Gaussian distributions is non-diagonal.The all covariance matrices in the Gaussian mixture model are non-diagonal.So the Variational auto-encoders based on the Gaussian mixture model is called non-diagonal Gaussian mixture variational auto-encoders(NDGMVAE).NDGMVAE allows the hidden variable z to more reliably match the hidden variable space.Further,in order to improve image resolution of generation from variational auto-encoders,we improved the structure of the encoder and decoder from variational auto-encoders,using the latest convolution neural network(CNN)and neural network with gating mechanism.We also compared the corresponding variational lower bound based on variational auto-encoders of different architectures.In order to prove that the newly introduced posterior distribution is more flexible and can match the hidden variables space more realistically,we experimented on MNIST,OMNIGLOT and Histopathology data,comparing the variational lower bound of marginal log-likelihood measured for each data set,and visualized MNIST,OMNIGLOT,and Freyfaces datasets.Moreover,we did the corresponding experiment based on the different numbers and the different coefficients of the Gaussian mixture model and the length of Normalizing Flows.In summary,the improved variational auto-encoders based on Gaussian mixture model had a significant improvement in performance and all kinds of application of variational inference,and has the advantage in theory.