An Information Theoretic Interpretation Of VAE And Its Applications

The Variational Auto-Encoder(VAE)is one of the most popular deep generative models in recent years.While enormous progress has been made to VAE models in the perspectives of both theoretical analysis and extensive application,the Degeneration in VAEs,which we address in this paper,yet exists and draws the attention in this domain.On one hand,the Degeneration shows that VAEs with deep networks suffer from the problem of representation learning and likelihood modeling,which seriously weakens the correlation between the encoding and decoding process,deviating from the goals of VAEs.On the other hand,the degeneration of VAE reflects the Trade-off between Representation Learning and Likelihood Modeling in VAEs,which has drawn wide attention in this domain,since VAEs are reported to face challenges in achieving these two goals simultaneously.Although several studies address these problems and plenty of variants of VAEs have been proposed to achieve the state-of-the-art performance in different tasks,due to the limited theoretical analysis,the solutions are too specific to be generalized to other application scenario.Thus,in this paper,we study VAEs in a view of information theory to give proper theoretical analysis and to propose corresponding solutions.To investigate how degeneration affects VAE from a theoretical perspective,we illustrate the information transmission in VAE and analyze the intermediate layers of the encoders/decoders.Specifically,we propose a Fisher Information measure for the layer-wise analysis.With such measure,we demonstrate that information loss is ineluctable in feed-forward networks and causes the degeneration in VAE.We show that skip connections in VAE enable the preservation of information without changing the model architecture.We name this class of VAE equipped with skip connections as SCVAE and perform a range of experiments to show its advantages in information preservation and degeneration mitigation.Moreover,SCVAE is demonstrated to be compatible with the state-of-the-arts.We also investigate VAEs in the Fisher-Shannon plane,and demonstrate that the representation learning and the log-likelihood estimation are intrinsically related to two information quantities: Fisher information and Shannon information.Through extensive qualitative and quantitative experiments,we provide with a better comprehension of VAEs in tasks such as high-resolution reconstruction,and representation learning in the perspective of Fisher information and Shannon information.We further propose a variant of VAEs,termed as Fisher auto-encoder(FAE),for practical needs to balance Fisher information and Shannon information.Our experimental results have demonstrated its promise in improving the reconstruction accuracy and avoiding the non-informative latent code as occurred in previous works.The experiments demonstrate the effectiveness of FAE in benefiting from the Fisher information in fitting data to achieve the fine-grained generation and the representation learning of high quality,or benefiting from the Shannon information in generalization to avoid problems like overfitting.