Research Of Machine Learning Techniques For Data-efficient Learning

In recent years,machine learning has gained great successes in diverse application areas,due to the increase of the data scale and the advance of the computing resources.However,the corresponding performance leaps heavily rely on massive amounts of labeled data,especially for deep learning.As deep neural network contains quite a large amount of parameters to be fitted,it will suffer from the problem of over-fitting when we do not have enough labeled training data.For many learning tasks,the cost of large-scale data annotation is prohibitively expensive.Data-efficient learning,which aims at improving the utilization efficiency of data and reducing the annotation burdens in machine learning tasks,has been becoming increasingly attractive over the past years.Data-efficient learning helps to greatly reduced the development cost of machine learning applications.Also,it can make machine learning techniques more practical,especially in data-poor domains.This thesis aims to implement data-efficient learning through both model adaptation and domain adaptation.Overall,its main content includes five parts.Specifically,the first three parts mainly focus on how to adaptively learn models,in which techniques like nonparametric Bayes or black box optimization are explored to adaptively select learning models according to the training data,in order to prevent the problem of over-fitting when we are faced with small-scale training data;On the other hand,the last two parts of this thesis mainly focus on data-efficient learning based on domain adaptation,which aims to relieve annotation burdens through transferring knowledge from other related data domains when faced with small-scale training data.The main contents of each part are detailed as follows:1.An effective generative classification model for multivariate categorical data is proposed.Specifically,it assumes that the multivariate categorical data are generated from a continuous latent space and adopts categorical latent Gaussian process to model their generative process.Through learning the continuous representation,it can overcome the problem of sparsity in categorical data and estimate the class-conditional probabilities more effectively.Due to the nonparametric Bayesian characteristic of Gaussian process,the proposed method can adaptively determine the learning model according to the training data and reduce the risk of over-fitting when the training data size is small.2.A discriminative categorical latent Gaussian process is proposed.In this method,all the training data are modeled together with a single latent Gaussian process,and the Fisher's discriminant is imposed over the posterior of the continuous latent space in order to learn a separable embedding.Combining the modeling ability of latent Gaussian process on scarce data with the discriminative ability of the Fisher's discriminant,the proposed method is expected to work well on classifying multivariate categorical data given only small-scale training data.3.An effective structure optimization method for neural networks based on estimation of distribution algorithm is proposed.Compared to Gaussian process,neural network can abstract the data layer by layer and obtain better representations of the data.However,as it contains a very large amount of parameters to be fitted,over-fitting usually appears when the training data size is small.In this method,estimation of distribution algorithm is adopted to adaptively learn the topical structures of neural networks according to the training data through estimating the distribution of the optimal solution.As a result,the final selected model can be made suitable for the current learning task and the problem of over-fitting can be relieved for data scare learining tasks.4.An effective domain adaptation method for image classification based on deep generative models is proposed.Specifically,the high-level layers of the source generator and the target generator are enforced to be tied,in order to control the labels of the generated target data.As a result,these generated target samples can be used to improve the classifier's discriminative ability on the target domain.5.An effective domain adaptation method for image segmentation based on structure enhancement is proposed.Compared to image classification,the annotation burden of segmentation tasks is more expensive.To overcome the data scarcity,this method aims to implement domain adaptation in semantic segmentation through enhancing the structure information of target images at both the featurelevel and the output-level of fully convolutional networks.However,limitations still exit in my current works,which motivates me to further focus on topics including deep Gaussian process,structure learning on complex tasks and domain adaptation based on structure learning in the future.