Structural Optimization Of Neural Networks Based On Binary Approach And Smooth Group L_1/2 Method

Artificial neural network is an imitation of the connection between biological neurons and several functions of the brain.It is established through appropriate mathematical descriptions,and different networks can be formed according to different connection methods.Because of its strong learning ability,large-scale data processing ability,and good generalization ability,neural network has become a research hotspot in many fields nowadays.In order to accelerate the computational speed of the network,to reduce the memory needed by the network,and to generate a more economical sparse network,the optimization of neural network structure has become a hot research topic.For the output layer,we first improve the traditional one-to-one approach(one-hot approach)and propose a binary approach.The novel approach can reduce the number of output nodes and hidden-output nodes,and speed up the computation.This idea is applied to perceptron network,feedforward neural network and extreme learning machine,and achieves good experimental results.Besides,for the hidden layer,the smooth group L_1/2 regularization method is used for the fully connected layer of the convolutional neural network.This smooth group L_1/2 regulariza-tion method can remove the redundant nodes of the fully connected layer,avoid the possible oscillation during the experiment,and improve the generalization ability of the network.The specific research is as follows:1.When a multi-layer linear perceptron network with four hidden nodes is used to deal with multi-classification problems,the traditional one-to-one approach costs too much output nodes.Therefore,this thesis proposes the binary approach for the output layer,instead of the one-to-one approach.Several separable definitions and theorems for the vertices of hypercube in n dimensional space are given,and then the hyperplanes that separate two classes of points are found.The combination of these hyperplanes can complete the multi-classification task.At the end,the feasibility and superiority of the binary approach are shown through algebraic calculation and numerical experiments.2.When a general feedforward neural network is used to solve multi-classification prob-lems,binary approach is applied for the output layer encoding.Due to the sigmoid function,we can only give proofs with the number of classes less than or equal to four.Some special cases are given to illustrate the advantages of the binary approach.The numerical experiment part com-pares the binary approach with the one-to-one approach on eight datasets.The binary approach has better experimental results on five out of the eight datasets and the calculation speed is faster than the one-to-one approach.3.When using the extreme learning machine to deal with multi-classification problems,the binary approach is employed to encode the output layer.We test the binary approach and the one-to-one approach on nine different datasets,and then compare the classification accuracy,prediction rate,recall rate,F1 measure,standard deviation and root mean square error of the two approaches.Compared with the one-to-one approach,the binary approach performs better on all criteria on seven out of the nine datasets.Moreover,the calculation speed of the binary approach is faster,and the selection of the hidden number has less influence on the experimental results.For the regularized extreme learning machine,we also conduct comparative experiments,which show that the binary approach has better classification accuracy than the one-to-one approach.4.At last,the thesis mainly studies the sparsity of fully connected layers for convolutional neural networks.Here,a smooth function is used to replace the absolute value function in the group L_1/2 method,and then a smooth group L_1/2 is obtained to prune the redundant nodes in the fully connected layer.In order to drive some weights connecting the redundant nodes of the fully connected layer and the output layer to zero,we introduce the smooth group L_1/2 regular term into the learning process of the network.The original L_1/2 regularization term is not smooth at the origin,which will lead to oscillation in the learning process.For the smooth group L_1/2 regularization method,the convergence theorem also can be proved.Compared with the group L_1/2 regularization method,the smooth group L_1/2 regularization method not only has better sparsity,generalization ability,but also improves the classification accuracy.