| Information-theoretic approaches to neural networks provide alternative ways of controlling network complexity through optimizing the mutual information or entropy of neural networks, based on Shannon's information theory. There are three primary approaches to training information-theoretic neural networks, i.e., the Infomax principle (maximizing mutual information), the maximum entropy method, and the cross-entropy method. Following Linsker's Infomax principle (1988), many studies have claimed that the Infomax principle is the basis of their theoretical foundation. However, a literature review shows that many studies of information-theoretic neural networks are vague about their theoretical foundation.;Therefore, this dissertation finds its motivation from the diversity of information-theoretic models in the literature and works to develop an information-theoretic model that is effective in promoting generalization. Based on a reinforced maximum entropy method, the Information-Equilibrium (Info-Eq) model is developed for training in a supervised, information-theoretic Multilayer Perceptron (MLP) neural network. Unlike general MLPs in which the hidden layer is modeled by the binomial structure (sigmoid function), the Info-Eq model employs the multinomial hidden structure (softmax model) for modeling the network hidden layer. This hidden structure is found to be effective in capturing the mutual information of the training data in which inherent inner structure exists, for example, in geographic information. The performance of this model has been shown to be capable of stabilizing the mutual information of the training data and promoting generalization performance. This model also permits stochastic training and time-saving advantages.;With the preliminary regression problems and the two large-scale case studies of classification and time series problems in the Geographic Information System (GIS) area, we show the contribution of this research in both the fields of neural networks and GIS, where the research: (1) clarifies the underlying properties of information-theoretic neural networks; (2) formalizes the Info-Eq model for training in a supervised, information-theoretic neural network; and (3) provides alternative regression and classification models for the geographic information process with non-linear and non-Gaussian properties. |
