Convergence Analysis Of Gradient Algorithms For Training Higher-Order Neural Networks

The traditional neural networks (e.g. multiple-layer perceptrons) are composed of multiple layers of summation units. These networks are widely used in various fields and attract many researchers' interests. Due to the summation structures, the lack of nonlinearity greatly restricts their application to complicated cases. For example, although these networks can effectively solve approximation problems and classification problems, a large number of summation units are required for the traditional feed-forward neural networks to approximate a complicated function with the high cost of networks and the poor generalization.In order to overcome the shortcomings, Higher-Order units such as Sigma-Pi units, Product units and Pi-Sigma units etc., are introduced to replaced summation units. The neural networks incorporating the Higher-Order units are called Higher-Order neural networks (HONNs) including Sigma-Pi neural networks (SPNNs), Pi-Sigma neural networks (PSNNs) and Product-Unit neural networks with exponential weights (PUNNs), etc. There is a lot of discussion to the performance and application of Higher-Order neural networks. However, there is less theoretical exploration, due to the complex structure of Higher-Order neural networks.The gradient algorithm is the most popular training algorithm for feed-forward neural networks . There are two different ways to implement the gradient algorithm: online version and batch version. One of the main works of this dissertation is the convergence analysis of gradient algorithms for training Higher-Order neural networks. The convergence results on Sigma-Pi neural networks and Product-Unit neural networks with exponential weights are given, respectively . Additionally, the realization to Boolean functions with Higher-Order neural networks is also considered. Some efficient schemes are presented to deal with the realization to Boolean functions.The organization of the dissertation is as follows. Some background information about Higher-Order neural networks and the gradient algorithm is reviewed in Chapter 1.The second chapter points out a unified convergence result of the batch gradient algorithm for SPNN learning is presented, covering three classes of SPNNs:Σ-Π-Σ,Σ-Σ-ΠandΣ-Π-Σ-Π. The monotonicity of the error function in the iteration is also guaranteed.In the third and the forth chapter, the convergence analysis of the batch gradient algorithm and the online gradient algorithm for training PUNNs are presented, respectively. The monotonicity of the error function in the training iteration process is also guaranteed. The results support the local behavior of the optimization combining global random search algorithm with local optimization (gradient algorithm). The corresponding numerical examples are given to support the theoretical findings.The realization to arbitrary Boolean functions is discussed in Chapter 5 and Chapter 6. Binary Product-Unit neural network (BPUNN) and Binary Pi-Sigma neural networks with input conversion (BPSNN) are introduced to compute Boolean functions in terms of principal disjunctive normal form (PDNF) and principal conjunctive normal form (PCNF), respectively.