Information Geometry Based Pure High-order Interaction Model And Its Applications

On learning the statistical model, we usually need to analyze the interaction between random variables. Based on the theory of information geometry, we define the pure interactions between multiple variables, and present a novel method for extracting the pure high-order interactions. And we also test our method on real-life text processing tasks.Information geometry is the study of probability by way of differential geometry. It considers a space of probability distribution as a manifold in a Riemannian space, and studies the properties of probability distribution by analyzing the structure of geometry object. In this paper, we present a mixture coordinate system which can represent the degree of pure interaction properly. And we provide a method of checking the pure high-order interactions, and then a theoretical basis on the setting of parameters of the model. What is more, we give the relation of implication between several kinds of high order interactions theoretically.In the text processing tasks, the presence of a word can be considered as a Boolean random variable. The naive model usually assumes that all the random variables are independent, which is unsuitable in many cases. Based on the theory of information geometry, we present an efficient algorithm employing sliding windows and bottom-up incremental approach for mining the high-order word patterns. We test our method on an extended vector space model for text classification on Reuters-21578 and 20 newsgroups datasets, and also test our method on improving the classical N-gram language model. The experimental results show that our algorithm is effective and practical.