A Study Of Boltzmann Machines For Classification And Ranking Tasks

Boltzmann Machine(BM) is a kind of stochastic graph model usually used for distribution estimation. This paper aims to study the novel adaptions and applications of BM in data classification and information retrieval. This paper proposes and analyzes a Rényi divergence based generalization for discriminative learning objective of Classification Restricted Boltzmann Machine(ClassRBM). Specifically, we extend the Conditional Log Likelihood(CLL) objective to a general learning criterion. We prove that, some existing popular training methods can be derived from this generalization, via adjusting the parameters to specific values. Moreover, we show that this generalized criterion actually extends the CLL objective with a Rényi divergence-based regularization. Besides, we can replace the uniform distribution used in this divergence-based regularization by some sample-based distribution and we call the appended loss as general margin. The proposed generalization enables an effective model selection procedure and experiments achieved significant performance improvement over the existing learning methods on data classification tasks. In information retrieval, we proposed a novel retrieval method making use of BM. We aim to generalize the multinomial distribution assumption in traditional language model by exploring the use of fully-visible Boltzmann Machines(BMs) for document modeling. BM is a stochastic recurrent network and is able to model the distribution of multi-dimensional variables. It yields a kind of Boltzmann distribution which is more general than multinomial distribution. We propose a Document Boltzmann Machine(DBM) that can naturally capture the intrinsic connections among terms and estimate query likelihood efficiently. We formally prove that under certain conditions(with 1-order parameters learnt only), DBM subsumes the traditional document language model. Its relations to other graphical models in IR, e.g., MRF model, are also discussed. Our experiments on the document reranking demonstrate the potential of the proposed DBM in information retrieval.