Dendritic Neural Network Optimization

A typical biological neuron receives synaptic inputs from proximity neurons on its dendrite tree and sends the efferent axonal output downstream.In artificial neural network models,dendrite trees are modeled as linear structures that funnel the weighted inputs information to the output of neurons.However,numerous studies have shown that dendritic structures are far more than just such simple linear accumulators.The dendritic structures in biological neurons are highly nonlinear,and synaptic inputs can have a nonlinear effect on their adjacent synapses.Such property greatly elevates the contribution of local nonlinear components in neuronal outputs and empower neuronal networks with much greater information processing ability.In this paper,we apply the active dendritic structure in biological neurons to the traditional artificial neural network,that is,to model the local nonlinearity of dendritic trees with our dendritic neural network(DENN)structure.The main research content of this paper is to explore the advantages and performance characteristics of DENNs on the information processing ability.The main research content of this paper can be summarized as follows:First,we explore the properties of DENNs from theoretical aspect.We first describe the structure and definition of the DENN and standard feedforward neural network(FNN)models separately in a comparative manner.Then we prove that similar to traditional FNNs,a DENN is also a universal approximator,and can approximate the objective function with arbitrary precision.We finally study the neural network expressivity by measuring the number of linear regions or transition events between linear regions in the function space.Experiments illustrate that DENN has high expressive power.Then,we analyze the advantages and performance of the DENN with piecewise linear activations in supervised machine learning tasks empirically.We find that the DENN has a higher fitting power than the standard FNN,especially when the network is small,when trained with the natural image datasets.We also empirically show that the locality structure of DENNs can improve the model generalization performance,DENNs outranking naive deep neural network architectures,when tested on the classification tasks from the UCI machine learning repository.