Regularization Methods For Pi-Sigma&Sigma-Pi-Sigma Neural Networks

In recent years,neural networks have been widely used for various kinds of regression and classification problems.Many regularization techniques have been proposed by adding a regularization term into the neural network training process.There are two typical regularization(penalty)terms,employing L2 norm and L1 or L1/2 norm respectively.The function of L2 norm is mainly to get bounded network weights and to improve the generalization capability of the network.The function of L1 or L1/2 norm is mainly to sparcify the network so as to use less number of nodes and weights of the neural network without causing real damage to the network efficiency.This thesis is concerned with the regularization methods for high order neural networks(HONNs).These HONNs have been proved to be more efficient than the ordinary(first or-der)neural network in some respects.The sparsification issue is somehow more important for HONNs,since there are usually more nodes and weights in HONNs.In particular,we consider pi-sigma neural networks(PSNNs)and sigma-pi-sigma neural networks(SPSNNs).The main contents of this thesis are listed as follows:1.In Chapter 2,the online gradient method with L2 inner-penalty for PSNNs is studied.Here the L2 norm is with respect to the input value of each Pi node.The monotonicity of the error function,the weight boundedness,and some weak and strong convergence theorems are proved.2.In Chapter 3,we describe another L2 inner-penalty for PSNNs.But now the L2 norm is with respect to each weight.The convergence of the batch gradient method is proved.The monotonicity of the error function with the penalty term during the training iter-ation process is proved,and the weight sequence is proved to be uniformly bounded.The algorithm is applied for solving four-dimensional parity problem and the Gabor function problem to support our theoretical findings.3.In Chapter 4,we proposes an offline gradient method with smoothing L1/2 regular-ization for learning and pruning of the PSNNs.The original L1/2 regularization term is not smooth at the origin,since it involves the absolute value function.This caus-es oscillation in the computation and difficulty in the convergence analysis.A smooth function is used to replace and approximate the absolute value function,ending up with a smoothing L1/2 regularization method for PSNNs.Numerical simulations show that the smoothing L1/2 regularization method eliminates the oscillation in computation and achieves better learning accuracy.We are also able to prove a convergence theorem for the proposed learning method.4.In Chapter 5,we consider the more important Sigma-Pi-Sigma neural networks(SP-SNNs).In the existing literature,in order to reduce the number of the Pi nodes in the Pi layer,a set of special kind of multinomials Ps are used in SPSNNs.Each multino-mial in Ps is linear with respect to each particular variable ?i when taking the other variables as constants.This choice may be somehow intuitive,but is not necessarily the best.We propose in this thesis an adaptive approach to find a better multinomial for a given problem.To elaborate,we start from a complete set of multinomials with a given order.Then we employ a regularization technique in the learning process for the given problem to reduce the number of monomials used in the multinomials,and end up with a new SPSNN involving the same number of monomials(= the number of nodes in the Pi-layer)as in Ps.Numerical experiments on some benchmark problems show that our new SPSNN behaves much better than the traditional SPSNN with Ps.