Research On Structure Design For Deep Neural Networks

In recent years,deep learning has a further development on the base of artificial intelligence and machine learning,which has gradually become the research focus of many well-known scholars and companies from all over the world.Moreover,deep learning has made satisfactory achievements in many areas of academic researches and practical applications.While,the structure design of deep neural networks is a basic problem of model training in deep learning,and it is also a very important factor for effectively fitting complex functions.What's more,designing structures of deep neural networks quickly and effectively plays a decisive role for easier training and better generalization.However,the problem of how to design the structures of deep neural networks has not been well solved at present,and a more effective method is needed to complete the tough task.To overcome the defects and deficiencies of the existing methods for structure design,this paper proposes the layer-wise PCA,growing layer-wise PCA and layer-wise PCA framework to design the structures of deep neural networks,consist of Deep Multiple-layer Perceptions,Deep Auto-Encoders,Deep Belief Networks and Deep Boltzmann Machines.The main research findings are described as follows:1.Propose the layer-wise PCA.This method can effectively design the structure of a deep multiple-layer perception when the number of hidden layers is certain.When given the training datasets,the number of hidden layers and the cumulative contribution rate threshold of PCA,the method can adaptively determine the number of neurons in each layer of a deep multiple-layer perception.And the details of design process are as follows: firstly,the number of input neurons is taken as the dimension of the training data;secondly,the number of neurons in the second layer is computed as a PCA dimension from the training data by appropriately controlling information loss;thirdly,the number of neurons in a layer between the second and the output layer are repeatedly computed from the activations of neurons in its previous layer followed by a PCA;finally,the number of output neurons is taken as the number of class labels.2.Propose the growing layer-wise PCA.This method can effectively design the structure of a deep multiple-layer perception when the number of hidden layers is uncertain.At first,gradually adjust the number of hidden layers in a certain range(usually ?10),and then use the layer-wise PCA to design the structure for different layers.Finally,sufficiently train these deep multiple-layer perceptions and using the validation dataset to validate their performance,output the superior structure and parameters.3.Propose the layer-wise PCA framework.The framework can effectively design structures of many deep neural networks according to the data distribution and model characteristics.In details,the deep neural networks contain Deep Multiple-layer Perceptions,Deep Auto-Encoders,Deep Belief Networks and Deep Boltzmann Machines.The experimental results show that the methods and the framework proposed in this paper can efficiently design the structures of multiple network models according to the dataset distribution and low information loss.The experiments strongly prove that the methods and framework can greatly reduce the number of neurons and training parameters,and save considerable computing time and convergence time.What's more,they can significantly decrease the difficulty of training networks,and enhance the ability of feature extraction,feature expression and generalization,which can build a firm foundation for wide applications of deep neural networks.