Approximation Capabilities Of Sum-of-Product Neural Networks And Radial Basis Function Neural Networks

In recent years, neural network theory has developed rapidly. Approximation theory of neural networks is important for analyzing the computation capability of neural networks. Mappingsin approximation applications are usually very complicated. Moreover, we can not expect to be able to compute exactly the unknown mappings. Thus, a current trend is to use artificial neural networks to approximate multivariate functions by computing superpositions and linear combinations of simple univariate functions. This is related to the density problem of neural networks: whether, or under what conditions, is a family of neural network output functions dense in a space of multivariate functions, i.e., approximation capability of neural networks. Approximation capability of neural networks, which is a basic problem in neural networks, has aroused extensive attention among engineers and mathematicians along with the development of neural networks. Density is the capability to approximate functions in theory, but denseness does not always give an effective scheme. A class of networks can not be used for approximationwithout guarantee of denseness. From a mathematical point of view, the approximation problem of neural networks can be studied from four aspects: function approximation, approximationof families of functions (strong approximation), functional approximation and operator approximation. Many neural network models have been proposed so far. The feedforward neuralnetworks are most widely used in applications, so it is important to study approximation capabilities of various feedforward neural networks.There have been deep investigations on approximation capability of radial basis function (RBF) neural networks. But the known results still need to be improved. Meanwhile, the approximation capability theorems of RBF neural networks and multilayer perceptron (MLP) neural networks are used in the investigations of their approximation capability to families of functions. Thus, we will ask: is there similar relationship between approximation of functions and family of functions for general feedforward neural networks? It is desirable to propose an integrated theoretical framework for the above problem. Sum-of-Product neural networks (SOPNN) and Sigma-Pi-Sigma neural networks (SPSNN) are proposed in 2000 and 2003, respectively. Product and additive neurons are their basic units. The new structures overcome the extensive memory requirement as well as the learning difficulty for MLP neural networks and RBF neural networks. They have novel performance in function approximation, prediction, classification and learning control. We discuss both the uniform and L^p approximation capabilities of them.In comparison with the conventional existence approach in approximation theory for neural networks, we follow a constructive approach to prove that one may simply randomly choose parameters of hidden units of three-layered Translation and Dilation Invariant (TDI) neural networks and RBF neural networks, and then adjust the weights between the hidden units and the output unit to make the networks approximate any function in L²(R^d) to arbitrary accuracy. Furthermore, the result we obtained also presents an automatic and efficient way to construct an incremental three-layered feedforward networks for function approximation in L²(R^d).Ridge functions in the form of g(a·x) and their linear combinations are widely used in applications on topology, neural networks, statistics, harmonic analysis and approximation theory, where g is a univariate function, and a·x denotes the inner product of a and x in Rⁿ. When we study a function represented as a sum of ridge functions, it is fundamental to understand to what extent the representation is unique. The known results consider two cases:g∈C(R)and g∈L_loc¹(R).We draw the same conclusion under the conditions g∈L_loc^p(R)(1≤p<∞),or g∈D'(R) and g(a·x)∈D'(Rⁿ).Provided that a function is represented by a sumof ridge functions, the relationship between the smoothness of the given function and the sum components is also analyzed.This thesis is organized as follows:Some background information about feedforward neural networks is reviewed and the significanceof approximation capability theory of neural networks is introduced in Chapter 1. The methods usually used and progress in researches on approximation capability theory of neural networks is also presented in this chapter.Investigated in Chapter 2 is the uniqueness of representation of a given function assome sum of ridge functions. It is shown that if f(x)=∑_i=1^m gi(aⁱ·x)=0,aⁱ=(a₁ⁱ,…,a_nⁱ)∈Rⁿï¼¼{0}are pairwise linearly independent, and gi∈L_loc^p(R)(or gi∈D'(R),gi(aⁱ·x)∈D'(Rⁿ)),then each g_i is a polynomial of degree at most m - 2. In addition, atheorem on the smoothness of linear combinations of ridge functions is also obtained.Chapter 3 mainly deals with capability of RBF neural networks to approximate functions, family of functions, functional and operators. Besides, we follow a general approach to obtain approximation capability theorem for feedforward neural networks to a compact set of functions.The results can cover all the existing results in this respect.It is proved in Chapter 4 that the set of functions that are generated by SOPNN with its activation function in C(R) is dense in C(K), if and only if the activation function is not a polynomial. The sufficient and necessary condition under which the set of functions generated by SPSNN is dense in C(K) is also derived. Here IK is a compact set in R^N.In Chapter 5, we give a sufficient and necessary condition under which the set of functions that are generated by SOPNN is dense in L^p(K). Based on the L^p approximation result of SOPNN, the L^p approximation capability of SPSNN is also studied.Chapter 6 studies approximation capability to L²(R^d) functions of three-layered incrementalconstructive feedforward neural networks with random hidden units. RBF neural networks and TDI neural networks are mainly discussed. Our result shows that given any non-zero activationg : R₊→R and g(‖x‖)∈L²(R^d) for RBF hidden units, or any non-zero activationfunction g(x)∈L²(R^d) for TDI hidden units, the incremental network output function with nrandomly generated hidden units converges to any target function in L²(R^d) with probability 1 as n→∞, if one only properly adjust the weights between the hidden units and output unit.