Deep Learning Models And Applications Based On The Restricted Boltzmann Machine

As a new model of training Multi-layer Neural Network with a strong power of representation and a wide range of uses, Deep Learning has induced attention of machine learning community. Restricted Boltzmann Machines(RBM) is a type of probabilistic random neural network model with two-layer architecture. There are fully connections in the two layers but no connections within the same layers. RBM is an effective method to detect features, which is initialized traditional feed-forward neural networks and its generalization capability is significantly improved. A Deep Belief Network(DBN) composed of several RBMs can detect more abstract features. Due to the advantages of RBM, this paper starts from its formula, inducing learning algorithm, parametric setting and convergence theories, mainly focuses on the following parts:1. Based on the RBM model, we study two different sparse RBM models: the Sp RBM and the Log Sum RBM. Combined with the advantage of Polyak Averaging to accelerate the convergence in stochastic gradient descent, the iterative algorithm of sparse RBM is improved. We analyze the differences and advantages beween mentioned models from the aspects of feature representations and algorithms' colplexity.2. The two assessment strategies of RBM(the reconstruction error and the annealed importance sampling) are applied to evaluate the sparse models. The experiments verify the feasibility and efficiency in detail and assess that the Log Sum RBM model is better than the Sp RBM model from the perspective evaluation.3. We study the influence of the numbers of DBN layers on the performance and obtain the optimal one, that is four. So the nonlinear network models with depth of four layers and five layers are constructed by using different models based on RBM to make pre-training respectively, such as 3DBN, 3Sp DBN, 3Log Sum DB, 4DBN, 4Sp DBN and 4Log Sum DBN. We conduct some experiments to achieve classification and recognition on MNIST and UCI data sets. By contrasting to the experimental results, we indicate that the Log Sum DBN based on Polyak Averaging is one of the optimal model because of its stronger power of sparse feature representations and discriminations, which coincides with the aboved evaluation results.