Research On Generalization Capability Of Autoencoder And Broad Network Based On Localized Stochastic Sensitivity

In machine learning,neural networks are expected to have small output perturbations with respect to small input perturbations,which indicates a high robustness and high generalization ability.However,autoencoders and broad networks are easily affected by input data with small perturbations or noise.In addition,usingunseen samples far from training samples to measure the generalization ability of the network may be misleading.It is unreasonable to calculate the generalization error of a network using the entire input space.Therefore,this thesis introduces the localized stochastic sensitivity(LSS).The LSS measures the expectation of squared output differences between training samples and unseen samples located within their -neighborhood for a given autoencoder or broad network.Then,a training method for autoencoder with high generalization is derived.The training method of autoencoder is further extended from both deep and broad perspectives,i.e.,a deep autoencoder and a broad network training methods based on localized stochastic sensitivity are proposed.These theoretical methods are proposed to improve the robustness and generalization ability of the network with respect to small input perturbations by minimizing the localized stochastic sensitivity.These provide important theoretical and technical support for the current research on the generalization ability of neural networks.The main research content and main contributions of this thesis are summarized as follows:(1)Autoencoders are sensitive to input with small perturbations,which leads to the problem of low robustness and low generalization capability.Therefore,this thesis proposes a Localized Stochastic Sensitive Autoencoder(LiSSA)algorithm based on localized stochastic sensitivity.By introducing localized stochastic sensitivity,the LiSSA reduces the sensitivity to unseen samples that are slightly different from training samples and improves the generalization ability of the network.Experimental results on thirty benchmark datasets show that the LiSSA is significantly better than several classical and state-of-the-art autoencoder algorithms in image reconstruction and classification tasks.In addition,the LiSSA also yields good performance on three Skeleton-based Human Action Recognition datasets.(2)This thesis extends the theory of localized stochastic sensitivity from the deep perspective and proposes a Deep Autoencoder with Localized Stochastic Sensitivity(D-LiSSA)algorithm.The D-LiSSA strengthens the generalization ability of the classification or forecasting model by minimizing the localized generalization error(including training mean square error and localized stochastic sensitivity)and learns information-rich hidden representations from unseen samples within -neighborhood of training sample.In addition,this thesis uses six benchmark image datasets and four real public electrical load data provided by the European power system ENTSO-E to evaluate the performance of the D-LiSSA.Experimental results with classic and latest models show that the D-LiSSA obtains the best results for image classification and real time series data forecasting tasks.These results valiate that the D-LiSSA yields good generalization ability and robustness.(3)The localized stochastic sensitivity theory is extended from the broad perspective.Broad networks' performance degrades when facing complex noisy environment.As such,this thesis proposes a Broad Network based on Localized Stochastic Sensitivity(BASS)algorithm by introducing the localized stochastic sensitivity to broad network,which enhances the robustness of the broad network to noisy and perturbed data.In addition,three incremental learning algorithms are provided.These ensure that the BASS can be quickly constructed when new samples arrive or the network is deemed to be expanded,without retraining the entire model.The BASS yields good performance for experiments on both regression and classification tasks.