A Method Of Improving Restricted Boltzmann Machines Via Theta Pure Dependency

With the development of deep learning,Deep Belief Networks(DBNs)have shown important impact on a wide range of signal and information processing work.Restricted Boltzmann Machines(RBMs),play a significant role in deep learning,serving as a learning block in deep neural networks(DNNs).A critical issue for RBMs remains model selection.Over fitting problem will be easily caused by inappropriate number of hidden units or too larger parameter space with increasing number of layers.In this paper,we introduce Pure Dependency-Restricted Boltzmann Machine(PD-RBM)to alleviate over fitting issue of deep neural networks.Compared to RBM,PD-RBM is able to automatically change its structure facing different data in order to make the model complexity fit data,which is beneficial to improve generalization ability.Specifically,we conduct pre-training on data to collect the statistical information among features.In the phase of constructing model,we design an algorithm in order to make use of these information to construct the structure of PD-RBM.The establishment of PD-RBM is based on Theta Pure Dependency(TPD)which is a statistical measure of variable dependency,defined under Information Geometry framework.Information geometry is the study of probability by way of differential geometry.It considers a space of probability distribution as a manifold in a Riemannian space,and studies the properties of probability distribution by analyzing the structure of geometry object.We evaluate the effectiveness of PD-RBMs on MNIST.In the phase of experiments,we design three different experiments to test PD-RBMs.Experimental results show that PD-RBMs significantly mitigate the overfitting problem and improve test performance.