Research And Applications Of Feature Representation Methods In Deep Learning

In recent years,with rapid development of the computer hardware,deep learning has achieved a great progress in the fields of computer vision,speech recognition,natural language processing and health care,and so on.Feature representation and learning is the most foundational and important problem in the study of deep learning.However,deep learning-based feature representation has various kinds of difficulties and problems such as lacking of generalization error bounds,over-fitting,under-fitting,vanishing gradient,exploding gradient,insufficient feature extraction,and so on.This thesis concentrates on the problem of feature representation in deep learning and carries on the studies of robustness of feature representation,training of deeper networks,sufficient feature extraction and application with missing value data.The main research work and innovation are presented in the following aspects.(1)The general properties of all algorithms that are based on traditional Auto-Encoders are summarized: Firstly,the generalization lower bound of reconstructing the input is presented.Then,it has been proved theoretically that the reconstruction error of the input can not be lower than this lower bound,which can be viewed as a guiding principle for reconstructing the input.Sencondly,the reconstruction of a hidden representation achieving its ideal situation is the necessary condition for the reconstruction of the input to reach the ideal state.Thirdly,minimizing the Frobenius norm of the Jacobian matrix of the hidden representation has a defciency.Minimizing reconstruction error of hidden representation has an ability to cope with this problem and is more robust for feature learning than minimizing the Frobenius norm of Jacobia matrix of hidden representation.Based on the above analysis,a new unsupervised feature representation method,DDAEs,is proposed,which uses corruption and reconstruction on both the input and the hidden representation to learn robust feature representation.Compared with other methods,DDAEs is much more flexible,extendible,antinoise,robust and accurate.Comparative experiments are carried out on UCI datasets,digit image recognition datasets and human genome sequence datasets which illustrate its effectiveness and robustness.(2)Considering the problems of training deeper networks such as difficulties of training,vanishing gradient,over-fitting,and so on,a novel structure termed cross-layer neurons architecture is presented,which has the ability to train deeper neural networks.It uses cross-layer neurons to collect and integrate the features learned from all the lower-level layers and sends them to the higher-level layers.With such architecture,higher-level layers can learn highly abstract features and have the potential to take the advantages of all lower-level layers.It is shown that cross-layer neurons architecture not only can relieve the problem of vanishing gradient,but also has the capability to improve the convergence rate of classification.Based on this novel architecture,a new feature representation method termed Cross-Layer Neurons Networks(CLNN)is proposed.Experiments are carried out on several benchmark datasets(e.g.,MNIST,CIFAR-10,CIFAR-100,SVHN and STL-10)which verifies the effectiveness of the proposed method.(3)In order to solve the problem of insufficient feature extraction,a new feature representation method based on multi-view is proposed.It firstly utilizes different views to study different characteristics of feature maps,and then merges them together to learn some union features.Considering that a full connection layer mixes the data together as a vector,wasting a lot of spatial information,a new feature representation method without full connection layer is presented.Comparative experiments on video-based person re-identification datasets(i LIDS-VID,PRID-2011 and MARS)demonstrate that the proposed feature representation methods are significantly better for feature extraction than the state-of-the-art models.(4)A locally auto-weighted least squares imputation(LAW-LSimpute)method is presented for missing value estimation,which can automatically weight the neighboring genes based on the importance of the genes.It firstly develops a quantitative model for every gene according to its importance,and then assigns an optimized weighting factor to each gene with Lagrangian method.In order to improve the convergence,a measure of uncertainty is introduced and an iterative missing value estimation method is designed,ILAW-LSimpute.Comparative experiments illustrate that the ILAW-LSimpute method can reduce the estimation error.Furthermore,to show the robustness of DDAEs and the effectiveness of ILAW-LSimpute for downstream analysis like classification,two more comparative experiments are conducted.The first one compares the classification results of different classification methods with different missing value estimation algorithms.The second one shows the relationships of classification accuracy between different classification methods and different missing value estimation algorithms with different missing rates.Experiments are carried out on gene expression datasets which verifies the robustness of DDAEs and the effectiveness of ILAW-LSimpute for classification.