Representation Ability Research Of Auto-encoders In Deep Learning

Feature, as raw material of machine learning systems, has a significantinfluence on the final model. The performance of machine learning systems dependslargely on the choice of feature representation. When data is expressed as featurewell, even a simple model can achieve satisfactory precision. So in a practicalapplication, an important step is to preprocess the data well and get a good featurerepresentation. In the image processing field, taking the original pixel values aslow-level feature representation often leads to a high dimension. It’s necessary todeal with high-dimensional feature with appropriate methods to get more conciseand effective feature representation. As a research hot topic, data dimensionalityreduction’s key idea is to represent the high-dimensional data effectively in alower-dimensional space, discover the interior structure of high-dimensional dataand then find a good and effective lower-dimensional feature representation. Thelow-dimensional feature not only keeps the most important information of originalhigh-dimensional data, but also makes follow up processing easier. This is a hot spotin the image processing field with great theoretical significance and practical value.Deep learning frameworks and unsupervised learning methods have becomeincreasingly popular and attracted the attention of many researchers in machinelearning and artificial intelligence fields. Recent research proved that deep learningmethods can indeed obtain very good results among the fields of image, audio andnatural language processing. Although deep learning methods have made muchprogress and been widely used, it is kind of like a black-box. There is no veryadequate and strict theoretical support and we have no clear understanding of why itperforms so well.Based on the discussion above, this paper starts from the building-blocks ofdeep learning methods and focus on the representation ability research, especially onthe ability to reduce dimensionality and the steadiness of the representation abilityof auto-encoders. The specific research contents of this paper are as follows:Firstly, in the direction of deep learning, people tend to make progress byemploying increasingly deep models and complex unsupervised learning algorithms.In this paper, we start from and focus on the building-blocks of deep learningmethods, i.e., single-layer auto-encoder (AE) and single-layer restricted Boltzmannmachine (RBM). Both methods can be seen as methods to transform representationand learn a new representation of the original data. And when we limit the numberof nodes in the second layer smaller than that in the first layer, we can also achievethe goal of dimensionality reduction at the same time. This paper expects to start from the basis of deep learning to understand it better.Secondly, principal component analysis (PCA), as a representative of lineardimensionality reduction methods, is a simple but widely used method. Meanwhile,auto-encoder and restricted Boltzmann machine can be seen as relatively newnonlinear dimensionality reduction methods. This paper attempts to betterunderstand single-layer auto-encoder and single-layer restricted Boltzmann machine.Since the main ideas of these two methods are similar and they mainly differ in thetraining process, we focus on the auto-encoder in the experiment part andinvestigate the representation ability of single-layer auto-encoder compared withprincipal component analysis.Thirdly, this paper highlights whether auto-encoder is a good representationtransformation method under the context of understanding visual features. Weevaluate the ability to reduce dimensionality and the steadiness of the representationability of auto-encoders based on original pixels and local descriptors against classicmethodologies like PCA. In the former part, we implement two kinds ofrepresentation transformation and a softmax classifier on MNIST dataset,investigate how the algorithm’s performance changes when the dimensionality of thetransformed representation varies, and evaluate the ability to reduce dimensionalityand the steadiness of the representation ability of auto-encoders based on originalpixels by the performance of the classifier. In the latter part, we evaluate the abilityto reduce dimensionality and the steadiness of the representation ability ofauto-encoders based on local descriptors by SIFT matching results. Experimentsbased on original pixels and local descriptors demonstrate auto-encoders’ ability toreduce dimensionality is better than principal component analysis. They alsodemonstrate the steadiness of the representation ability of auto-encoders and theeffectiveness and steadiness of the proposed AE-based transformation strategy.We actually tried several other ways to evaluate the representation ability ofauto-encoders, and investigated several variants of auto-encoder in order to obtainbetter performance but failed. This may indicate that single-layer auto-encoders andsimple non-linear transformation don’t have enough power to model the highdimensional data that well and that’s why we use deep models now. Finally, wesummarize the thesis of this paper and discuss future research direction.