Research And Implementation Of Fault-tolerant Technologies For Quantized Neural Networks

In recent years,with the continuous expansion of deep learning applications,neural network models have become increasingly popular in mobile devices and edge devices.However,due to factors such as hardware constraints and energy consumption constraints,these devices often cannot support high-precision computing,so it is necessary to use low-precision models to achieve efficient computing.When deploying neural networks on embedded devices,there may be various types of failures,including random errors caused by electromagnetic radiation,signal interference,voltage instability,aging,temperature changes,and other factors.On the other hand,errors caused by malicious attackers implementing hammer attacks on neural networks can lead to decreased accuracy or unavailability of neural network models,posing a security threat to intelligent applications.Therefore,the research on fault-tolerant technologies of quantized neural networks is crucial and has become a hot research topic.This thesis proposes a fault tolerance method of quantized neural networks based on architecture search to aim at hardware errors such as memory stuck up,timing disorder,and computational failures on embedded devices.Firstly,the possible types of hardware failures in the deployment of a quantized neural network model are analyzed.By studying the impact of two types of failures on the calculation process of the quantized neural network model,they are formalized.Then,these two types of errors are incorporated into the optimization target algorithm of neural network architecture search as constraints,and fault-tolerant training is added during the search process to obtain a quantized neural network model with enhanced fault-tolerance.Experiments of VGG-16,Resnet-18,and Mobile Net-V2 models are conducted on the CIFAR-10 dataset.The model accuracies of Mobile Net-V2 under two different fault mode settings are 68.0% and 35.6%,respectively.The fault-tolerant model accuracies obtained by our method are 86.5% and 79.3%,respectively,proving that the quantized neural network fault-tolerant model obtained by our method has high fault-tolerant performance.Then,to address the problem that bit flipping at different locations in the neural network has different effects on model accuracy,this thesis proposes a vulnerableparameter-aware fault tolerance method of quantized neural networks.This method explores the interpretability of the model,and identifies the vulnerable parameters locations of quantized neural networks based on parameter gradient range distribution,and then performs the enhanced model training based on parameter vulnerability to protect the parameters.Gradient concealment and range averaging reinforcement are performed on vulnerable parameters to mitigate the decline in inference accuracy caused by bit flip attacks on key parameters.On the CIFAR-10 dataset,experiments are conducted under two scenarios: random bit flip errors and malicious bit flip attacks.In the random bit flip error scenarios,the accuracy of the original model was 40.1% whereas that of the fault-tolerance enhanced model is 68.6%.In the malicious bit flip attack scenarios,the accuracy of the fault-tolerance enhanced model is still 70.9% after 10 rounds of iteration,while the accuracy of the original model is only 39.5%.It demonstrates that our method can effectively improve the fault-tolerant ability of quantized neural networks by protecting vulnerable parameters.Finally,based on the two previous methods,this thesis designs and implements a fault-tolerant system of quantized neural networks,including the functions such as faulttolerant model search,vulnerable parameter protection and error fault injection.The front-end provides modules such as fault-tolerant model search and training applications,while the back-end performs training and fault-tolerant protection based on the needs of the front-end,and returns model structure files and parameter files to the front-end,which can automatically generate the fault-tolerant enhanced models of quantized neural networks to users.