Research On Deep Neural Networks Based On RBM

A Restricted Boltzmann Machine(RBM)is an undirected graph model,which can be traced back to the Boltzmann distribution in Statistical Mechanics.As a neural network,the RBM has been widely used in machine learning tasks.However,the traditional RBM is subject to its expansibility and expression ability.This paper proposes some feasible methods to alleviate these problems and generalizes RBM theory from the following four aspects:(1)Build an RBM model with real-valued hidden units and extend its objective function;(2)Alleviate the over-fitting problem in the RBM for both pattern recognition and image generation;(3)Propose effective discrimative neural nets based on deep features embedded;(4)Propose deep generative models based on the RBM.Details are listed as follows:(1)Research on feature representation of the RBMIn the RBM,although binary hidden units simplify the feature representation,they also bring information loss of input data.In order to alleviate this problem,this paper expands binary units to real-valued units,and proposes a Gaussian RBM with auxiliary units(ARBM)for both pattern recognition and the image generation;For the image generation task,this paper extends the KL divergence to the adversarial loss,and proposes an Adversarial RBM based on implicit gradients(AIRBM).Based on the Gibbs sampling,the proposed AIRBM has implicit gradients,and BP algorithm can be directly used for its training,moreover,the sampling process in an AIRBM does not need an additional Markov chain.(2)Alleviate the over-fitting problem in the RBMBased on the Dropout method,the Dropout RBM and Dropout neural networks are widely used in pattern recognition.However,Dropout RBM does not perform well on some datasets in the image reconstruction task.In order to alleviate the over-fitting problem in the RBM for both image reconstruction and pattern recognition tasks,this paper treats the trainable parameters in an RBM as stochastic variables which are assumed to follow Gaussian distribution,and proposes a Weight uncertainty RBM(WRBM).The trainable parameters in a WRBM are the expectation and covariance of the Weight Gaussian distribution,and the weight used in each forward propagation is sampled from the parameterized Weight Gaussian distribution.(3)Discrimitive neural networks based on deep features embeddedIn a deep discrimitive model such as a Deep Belief Network(DBN)or a Deep Boltzmann Machine(DBM),the training process can be divided into two stages: an iterative pre-training stage and a global fine-tuning training stage.However,the global fine-tuning results depends on the pre-training.In order to avoid the complex training process,this paper proposes a deep discrimitive model which uses a kernel Extreme Learning Machine(KELM)as the classifier of a DBN model to avoid the global parameter fine-tuning.However,the features obtained by the unsupervised pre-training may be uncorrelated to the discrimitive task.In order to make full use of the deep features extracted in the pre-training,this paper proposes an Incremental Extreme Learning Machine(ELM)based on the deep feature embedded(ELM-DFE),which uses deep features as the implicit features of an ELM;By gradually adjusting the deep embedding features,the ELM-DFE will converge to its local optimal solution.(4)The application and development of the RBM for deep generative modelsThe training process of a DBM is complex.In order to effectively generate images and simplify the training process,this paper proposes an Adversarial Deep Boltzmann Machine(ADBM).In order to effectively generate color images,this paper introduces convolutional layers to an ADBM and proposes an Adversarial Convolutional Hybrid Deep Generative Net(AHDGN).In order to expand the RBM model,this paper uses the Boltzmann priori instead of a nonparametric priori in the Flow model,and proposes an Adversarial Non-Volume Preserving Flow model with Boltzmann priors(ANVPB).The effectiveness of the proposed models is verified in the image generation task.This paper has 33 figures,24 tables,and 172 references.