Structure Selection And Fine-tuning Of Deep Neural Networks Based On Data Complexity

Deep learning has been applied in many applications with satisfying performance recently.However,the structure design of deep neural networks still faces huge challenges.Currently,the structure is selected according to accuracy or prior knowledge,which is time-consuming and relies on professional knowledge.Stacked autoencoder(SAE)is one of the most popular deep learning models.The current prior knowledge based structure selection methods cannot be applied to SAE due to its two learning stages.In this study,a data complexity based structure selection method for SAE is proposed.Data complexity measures,which quantify the difficulty of separating the samples in a classification problem,are applied to evaluate the discriminant ability of the feature space learnt in SAE.We firstly verify that structure of SAE with lower data complexity obtain better generalization ability.An effective data complexity based method to select the structure of SAE in the unsupervised pre-training stage is proposed.Moreover,the data complexity is used as a regularization term in the objective function for the supervised fine-tuning stage.Experimental results illustrate the effectiveness of the proposed methods in the structure selection and optimization of SAE.Fine-tuning the structure of existing models is able to reduce the storage and improve performance.Compressing and pruning the parameters are usually applied to large-scale deep learning model,but the time complexity is high.Visualization of latent features is also an effective strategy to fine-tune the structure for image recognition model.However,this method does not guarantee accurate performance and only can be applied to image related applications.In this study,we firstly investigate the relation between data complexity and discriminant ability of the latent feature of deep learning.A structure fine-tuning method for fully connected network(FCN)and convolutional neural network(CNN)based on data complexity is proposed in the click-through rate prediction of online advertising.The architecture of the network is fine-tuned according to the data complexity of the feature space.The experimental results indicate the efficiency of proposed methods.