Variational Learning Algorithms Of Bayesian Neural Network And Its Application

Bayesian neural networks(BNNs)provide a probabilistic interpretation of deep learning by treating the network weights as random variables.BNNs have recently drawn much attention for their ability to avoid overfitting in the small dataset and provide well-calibrated posterior uncertainties.Variational BNNs perform variational inference over weights,but it is difficult to specify meaningful prior distributions and approximate posterior distribution in a high-dimensional weight space,and BNNs have a large number of model parameters,huge computational costs,difficult to train and vulnerable to adversarial examples.These problems limit the practicability and scalability of BNNs.In this thesis,we focus on developing novel variational BNN models,three variational learning methods are proposed to improve the prediction performance and uncertainty estimation of the model,and a multi-task dversarial training method is proposed to improve the robustness of the variational BNN,and the models are applied to the tasks of out of distribution data detection,active learning,skin lesions classification and medical image segmentation.The main work of this thesis can be summarised as follows:(1)In variational BNNs,Gaussian prior is easy to underfitting and leads to over-inflated prediction uncertainty.To address this issue,a variational sparse BNN with a hybrid prior is proposed.Specifically,the prior of the weights in the first H layers of the BNN is given by a regularised horseshoe prior,and the prior of the weights in other layers is given by a Gaussian scale mixture prior.Employing the regularised horseshoe prior to improve the generalisation and select features,a Gaussian scale mixture prior was used to prevent overfitting and improve the uncertainty estimation.The model allows for computationally efficient optimisation via stochastic variational inference.A series of experiments showed that the model offered competitive predictive abilities and reasonable posterior weight uncertainty estimations on non-linear regression,image classification,anomaly detection,and active learning tasks compared with state-of-the-art methods.(2)In variational BNNs,the variance converges to zero when Gaussian distribution is used as an approximate posterior distribution.To address this issue,a variational BNN based on Gaussian approximate distribution is proposed.This method uses the approximate Gaussian distribution as the approximate posterior distribution,which samples the mean of the Gaussian distribution through a normal distribution and the variance through an inverse gamma distribution.The prior distribution is the same as the approximate posterior distribution.Based on the variational inference technique,the approximate variational objective function is derived and the convergence of the model is proved.Extensive experiments on benchmark datasets show that the proposed method can achieve good prediction performance and reasonable uncertainty estimates.(3)BNNs need multiple forward passes in the prediction process and the variational loss can not always learn the best variational parameters.To address this issue,a scalable functional variational BNN with Gaussian processes(GPs)is proposed.Specifically,we regard the prior and variational posterior distributions as GPs,and perform variational inference in function space.We employ a variant of the functional evidence lower bound(fELBO)with a β-weight on the Kullback-Leibler divergence term as the loss objective,and design a content-aware UNet segmentation network,which utilizes content-aware reassembly of features(CARAFE)as an upsampling operator to extract semantic information from input feature maps.The proposed BNN framework enables efficient training and can perform predictive inference with one forward pass.Extensive experiments are conducted on four segmentation datasets;the experimental results demonstrate that the proposed approach improves the segmentation performance metrics and uncertainty estimates compared with several state-of-the-art methods.(4)BNNs are vulnerable to adversarial examples crafted by adding small,human imperceptible perturbations to natural examples.To address this issue,a novel multi-task adversarial training approach for improving the adversarial robustness of variational BNN is proposed.Specifically,we first generate stronger and more diverse adversarial examples for adversarial training by maximising a multi-task loss.This multi-task loss is a combination of the unsupervised feature scattering loss and supervised margin loss.Then,we find the model parameters by minimising another multi-task loss composed of the feature loss and variational inference loss.The feature loss is defined based on distance lp,which measures the difference between the two feature representations extracted from the clean and adversarial examples.Minimising the feature loss improves the feature similarity and helps the model learn more robust features,resulting in enhanced robustness.Extensive experiments are conducted on four benchmark datasets in white-box and black-box attack scenarios.The experimental results demonstrate that the proposed approach significantly improves the adversarial robustness compared with several state-of-the-art defence methods.