Adaptive Cardinality Restricted Boltzmann Machines

Recently, deep learning has been widely studied and a variety of achievements have been got. One of the most important reasons is using Restricted Boltzmann Machines(RBM) as building blocks of deep neural networks. There are lots of researches for improving the performance of RBM, one of which is Cardinality Restricted Boltzmann Machine(CaRBM). It is a RBM-based model which induces sparsity directly to the joint probability distribution of RBM. However, Ca RBM assumes that all input data share a fixed threshold of hidden unit activation, whereas it is a rare case in most of time.To solve the problem, in this paper, a framework based Ca RBM named Adaptive Cardinality Restricted Boltzmann Machine(AC-RBM) was proposed. In AC-RBM:(1) There is no the fixed-threshold assumption. Input data havetheir own thresholds of hidden unit activation. And thresholds of all input data can be modeled by a certain distribution that adapts to input data.(2) The learning algorithm is independent of the distribution. If the distribution is replaced, the only modification is using anothercorresponding sampling process instead.Contributions in this paper can be summarized as follows:1. The AC-RBM is proposed, in which an appropriate distribution is used to model thresholds of hidden unit activation.2. It is proved that the distribution in AC-RBM can be replaced. Also, the principle of choosing an appropriate distribution is illustrated.3. The Gaussian distribution case named Gaussian Cardinality Restricted Boltzmann Machine(GC-RBM)is elaborated. The reasons for choosing Gaussian distribution and the learning algorithm are given as well.4. An image classification system mainly based GC-RBM is implemented.5. The experiments are conducted on two real world datasets. Experimental results show that the GC-RBM outperforms the Ca RBM. Moreover, the limitation of GC-RBM is analyzed.