Restricted Boltzmann Machines: A Collaborative Filtering Perspective

A Restricted Boltzmann Machine is a particular type of Markov random field that hasa two-layer architecture. Recently, RBM becomes increasingly popular because of its fastlearning algorithm, Contrastive Divergence. For the theoretic perspective, the success ofRBM has greatly encouraged the research about the stochastic approximation theory, energy-based models, unnormalized statistical models. For the application perspective, RBM hasbeen successfully applied in various machine learning domains, such as classification, re-gression, dimension reduction, high-dimensional time series modeling, sparse overcompleterepresentations, image transformations, collaborative filtering.Based on a collaborative filtering perspective we build the connections between RBMand classical user-based algorithm and try to disclose the intrinsic mechanism of RBM andto improve RBM. In detail, this thesis includes the following three aspects:1. Based on classical user-based collaborative filtering algorithm, we introduce the con-cept of local user similarity and global user similarity. Local user similarity can be seenas a variant of tf-idf weight and try to emphasize the importance of the rates which areunusual for computing user similarity. Global user similarity can be seen as an appli-cation from spectral graph theory and takes the data sparsity problem in considerationby propagating similarity measurement. Based on both of Local User Similarity andGlobal User Similarity, we develop a collaborative filtering framework called LS& GS.An empirical study using the MovieLens dataset shows that our proposed frameworkoutperforms other state-of-the-art collaborative filtering algorithms.2. Restricted Boltzmann Machines are commonly used in unsupervised learning to ex-tract features from training data. Since these features are learned for regeneratingtraining data a classifier based on them has to be trained. If only a few of the learned features are discriminative, other non-discriminative features will distract the classi-fier during the training process and thus waste computing resources for testing. In thispaper, we present a hybrid third-order Restricted Boltzmann Machine in which class-relevant features (for recognizing) and class-irrelevant features (for generating only)are learned simultaneously. As the classification task uses only the class-relevant fea-tures, the test itself becomes very fast. We show that class-irrelevant features helpclass-relevant features to focus on the recognition task and introduce useful regular-ization effects to reduce the norms of class-relevant features. Thus there is no need touse weight-decay for the parameters of this model.3. Since learning in Boltzmann machines is typically quite slow, there is a need to restrictconnections within hidden layers. However, the resulting states of hidden units exhibitstatistical dependencies. Based on this observation, we propose using l1/l2 regulariza-tion upon the activation possibilities of hidden units in restricted Boltzmann machinesto capture the local dependencies among hidden units. This regularization not onlyencourages hidden units of many groups to be inactive given observed data but alsomakes hidden units within a group compete with each other for modeling observeddata. Thus, the l1/l2 regularization on RBMs yields sparsity at both the group and thehidden unit levels. We call RBMs trained with the regularizer sparse group RBMs(SGRBMs). The proposed SGRBMs are applied to model patches of natural images,handwritten digits and OCR English letters. Then to emphasize that SGRBMs canlearn more discriminative features we applied SGRBMs to pretrain deep networks forclassification tasks. Furthermore, we illustrate the regularizer can also be applied todeep Boltzmann machines, which lead to sparse group deep Boltzmann machines.