Research On Data Augmentation And Optimization Method For Deep Neural Network

As the future development of automobile,autonomous driving has a profound impact on automobile industry and transportation industry.The whole autonomous driving system is composed of a series of perception,planning,decision-making and control subsystems.Among them,the environment perception module is mainly responsible for obtaining driving environment information,generally including cameras,millimeter wave radar,lidar and othe r sensors.The most widely used and informative camera is the "eye" of the autonomous driving car.Computer vision technology is the basic technology of environment perception based on camera,and deep convolution neural network is the main tool in the fie ld of computer vision.However,due to the complex driving environment,there are still many challenges in computer vision of autonomous driving,and the accuracy of classification,positioning,segmentation and other tasks needs to be improved.On the one hand,the deep neural network model training needs a lot of training data,and the data acquisition used in autonomous driving is often costly and limited,which limits the generalization ability of large-scale deep neural network;on the other hand,the complex driverless environment requires more efficient training of deep neural network,and makes the model generalization ability stronger to adapt to variable driving environment.As the basic technology of visual perception technology,image classificat ion task plays an important role in semantic analysis of driving environment.Therefore,it is of great theoretical and practical significance to study the deep neural network based on image classification task.With the increase in network depth or network architecture optimization,the performance of DCNN methods has been boosted significantly.However,the performance improvement through network architecture optimization is approaching its limitation.More and more researches focus on other aspects of deep neural network.The training strategy of deep neural network has an important influence on the ultimate generalization capability of the network model.Deep neural network training strategy includes data preprocessing,loss function design,optimization method,model postprocessing approaches and so on.Therefore,compared with further improving the network depth or network structure,improving the training strategy of deep neural network is a more promising way to boost the generalization capability.Data augmentation algorithm is an important part of data preprocessing.Based on the limited training samples,data augmentation technology can generate new training samples,so as to increase the size of training dataset.In theory,based on the distribution of the original training data,the data augmentation method defines the vicinity of the original data,and takes new training samples in the vicinity to achieve augmented data.After the data preprocessing,the objective function should be drawn up,and the weights of the model should be adjusted by the optimization algorithm to achieve the minimum loss of the objective function.Among many optimization methods,gradient-based optimization method plays a leading role in the current deep neural network training,but there are still many problems,including oscillation,slow convergence and so on.The full-connected layer of deep neural network can be regarded as a linear system,and the least square method can be used to obtain the approximate minimum error.Therefore,the application of the least square solution to the optimization method can be explored.Moore-Penrose inverse can be used to calculate the least square solution,and because its solution not only reaches the minimum error,but also the norm of the solution itself is the smallest,which meets the requirements of the small weight norm of the depth neural network,it is very suitable for the training of the depth neural network.This dissertation focuses on the training strategy of deep neural network,including data angmentation,optimization algorithm,model post-processing algorithm.Moore-Penrose inverse is introduced into the training strategy of deep neural network.The specific research focus are as follows:1)A new data augmentation method is proposed.The method reinvent the purpose of the SLFN-based autoencoder by utilizing its decoding weight to explore the distribution neigh-borhood between the training samples of the same cate-gory.Moreover,the activated decoding weights are combined with original training samples from different classes in order to expand the vicinity t o the inter-class area.The distribution vicinity of the original training samples defined by our method is much broader than those by other data augmentation techniques,which results in betterlearned features and thus better classification accuracy.In experiments based on a large number of data sets and deep neural network models,this method has significantly improved the generalization performance of the model.2)A new post-processing method of the model is proposed.The method utilize the Moore-Penrose Inverse strategy to pull back the current residual error of the network to each fully-connected layer one by one,generating a desired output adjustment for each fully connected(FC)layer.Then according to the obtained desired output and input features,the approach uses the same strategy to recalculate weights in each FC layer.The method only recalculates the parameters in the fully-connected layers but never involves any network structure modification,which makes the proposed method fit for all existing DCNN models.3)An optimization algorithm for the top dense layer is proposed.This method is not dominated by gradient direction,but is to calculate the d ifference between the current weights and the optimal weights,which is defined as the optimal compensation.First,the weight is updated along the direction of the optimal compensation,and then the gradient is used to fine tune.In the experiment,compar ed with other optimization methods,this method has faster convergence speed and lower convergence loss;accordingly,the model trained by this method has a great advantage in classification accuracy.In addition,the optimal compensation and the convergen ce of the optimization method are proved mathematically.4)A gradient-free optimization algorithm based on optimal compensation is proposed.In this method,Moore-Penrose inverse is used to calculate the weight update in each training iteration,and the difference between the current weight and the optimal weight,i.e.the optimal compensation,is used to guide the weight update,which does not involve the gradient to the weight update.Experimental results show that the algorithm can achieve higher test accuracy than the gradient based optimization method when the number of trainable weights is the same.In addition,the method can adopt high learning rate and ensure convergence,so it has high practical value.5)The proposed methods are applied to the field of autonomous driving.The data augmentation algorithm is applied to the fine-grained vehicle classification task.It can effectively reduce over fitting and help to solve the problem of insufficient training data scale in autonomous driving application.The optimization algorithm based on the optimal compensation and gradient is applied to the fine-grained vehicle classification task,and the classification accuracy is greatly improved under the same model.The optimization algorithm based on the optim al compensation is applied to the classification of domestic traffic signs,and the classification accuracy is greatly improved in the transfer learning learning settings.The algorithms proposed in this paper have great theoretical and practical significa nce for improving the performance of deep learning related applications in autonomous driving systems.