Font Size: a A A

The Research In Deep Learning Model Based On Regularization

Posted on:2014-02-18Degree:MasterType:Thesis
Country:ChinaCandidate:X Y QuFull Text:PDF
GTID:2268330422964531Subject:Software engineering
Abstract/Summary:PDF Full Text Request
Motivated by the deep architecture in man’s brain, many researchers had tried to train artificial neural network having many layers(more than three layers), but they failed. In2006, The research in deep architecture had a breakthrough:Professor Hinton et al. in Toronto University proposed a model called DBN(Deep Belief Net) based on greedy layer-wise pre-training. This model could prevent the training of deep model from local problem.Then, many models based on RBM(Restricted Boltzmann Machine) or auto-encoder were proposed. These deep models had successfully been applied in many fields,such as character recognition, acoustic recognition, text classification, information retrieval and so on. As the RBM model could produce better representation than raw input, most of the deep models were based on RBM. Recently, some researchers have proved that RBM could be a standalone classifier. As a standalone model, it have successfully applied in many field.The RBM used for classification can be viewed as a three-layer neural network having input layer, hidden layer and output layer. The process of RBM training can be viewed as the process of dimension reduction. While we reduce the dimension,we are not sure that whether the structure in the input space is varied or not. Based on the smoothness assumption of supervised learning, we incorporate the regularization into the RBM model to make the process of RBM training more smooth. Recently, using unlabeled data to help supervised learning have become a research hot spot,we will combine RBM and EM to address the problem that the labeled data is too small. At the same time, we combine Statistical learning and Manifold learning to derive a RBM model based on semi-supervised learning. In other words, we introduce a Laplacian regularization into the optimization equation of RBM to form a new model called Laplacian Regularized RBM. The key point to train deep architecture was the unsupervised pre-training so far. However, most of unsupervised learning models were based on RBM, so the research in RBM is meaningful. In the process of training RBM,they reduce the dimension ignoring the structure in the data space, so introducing the regularization into RBM can address this problem. At the same time, this solution can alleviate the overfit problem.
Keywords/Search Tags:Deep learning, Restricted Boltzmann Machine, Semi-Supervised learning, Regularization, Manifold learning
PDF Full Text Request
Related items