Research And Implementation Of High-Performance Binary Convolutional Neural Network

The deep learning algorithms that have attracted extensive attention and research in the past ten years have achieved excellent performance in many fields including computer vision,and speech recognition.With the development of edge computing and embedded devices,it is more and more necessary to deploy deep learning algorithms directly on the edge devices in consideration of multiple aspects,such as data security and real-time computing.The high computing and storage resource requirements of deep learning algorithms contradict the limited resources of embedded devices,so deep learning models usually cannot be directly deployed on resource-constrained embedded devices.Binarization,as an extreme case of model quantification,can greatly reduce the storage space of model and increase the calculation efficiency,but there is a problem of the sharp decline in performance.We aim to study high-performance binary convolutional neural networks.On the one hand,we propose efficient solutions to several problems that affect the accuracy and speed of binary convolutional neural networks.On the other hand,we use neural architecture search technology to automatically generate high-performance binary convolutional neural networks that meet specific needs,and promote the deployment of deep learning algorithms on embedded devices.On the one hand,we propose efficient solutions based on experience to improve the accuracy and speed of binary convolutional neural networks.For the gradient mismatch problem that affects the accuracy of the binary convolutional neural network,we propose a new gradient approximation scheme,which can effectively improve the approximation accuracy of the back-propagation gradient.Experiments on the CIFAR10 dataset show that compared with the Htanh,which is widely used in the binarization algorithm,the proposed scheme has an accuracy increase of 1.47% to3.77%,and compared with Approx Func,the accuracy of the scheme is increased by1.79%;Since floating-point operation affects the computational speed of binary convolutional neural networks,we propose an efficient first-layer quantization algorithm,combined with the floating-point factor approximation strategy,to reduce residual floating-point operations in binary convolutional neural networks.Experiments on binary Vggsmall show that when 98% of floating-point operations are converted,the prediction accuracy of the model only drops by 0.14%;Finally,we implemented binary Vggsmall's inference process based on bitwise operations on the GPU platform,and verified the speed advantage of bitwise operations.On the other hand,we propose the Binary Neural Architecture Attack Defensive Search framework to explore the automated design of high-performance binary convolutional neural networks.The framework combines neural architecture search technology,adversarial attacks and defenses technology to automatically design a binary convolutional neural network with high parameter efficiency and high attack resistance.Compared with traditional differentiable neural architecture search algorithms,this framework has the following innovations: 1)More suitable search space for binary networks.The accuracy of target model based on this search space is41% higher than that of DARTS's search space;2)Efficient search strategy.We combine multi-stage search and stable search technology to significantly reduce the memory usage during the search process,and reduce the performance variance of the target model;3)Flexible search targets.We propose a customized cost constraint strategy,which can effectively weigh the performance and cost of the target model;4)Adversarial defense search.Based on the above improvements,we embed the adversarial defense training into the structure search process,focusing on finding a binary convolutional neural network with excellent anti-attack.The final target model searched by the BAADS algorithm is BAADSv3,which achieved 91.06%(CIFAR10)prediction accuracy with 1.95 M parameters,and 61.5%(Image Net)prediction accuracy with 6.9M parameters.Experiments have proved the excellent performance of BAADSv3 in terms of parameters,accuracy and anti-attack ability.