Research On Binary Quantization Algorithms For Deep Convolutional Neural Networks

Binary quantization is a crucial technique in model compression as well as an extreme form of model quantization,aiming to maintain the performance of deep convolutional neural network tasks while improving the model’s efficiency and memory utilization.To obtain more effective feature representations,convolutional neural networks adopt deeper and wider network structures,which increase the computation and parameter complexity of the network.To address this gap,model quantization has emerged as a research hotspot in recent years.Among the various methods,binary quantization has attracted significant attention due to its high compression capability.Binary quantized networks,also known as binary neural networks,can significantly speed up the inference process of models.Therefore,binary neural networks have a specific potential in model compression and optimization acceleration.However,binary networks convert weights and activation values from 32 bits to 1 bit,leading to an information loss in the process.The current methods for network weights and activation do not adequately extract the required information.Focusing on the problem of insufficient information extraction,which is prevalent in current binary quantization,our work conducts in-depth analysis and research as follows:(1)We propose a binary quantifying method of neural network based on W regularization and variant cosine momentum.We introduce a regularization method suitable for binary quantization under the framework of deep learning to address the problem of insufficient extraction of weight information.Traditional binary quantization typically applies L1 normalization to the data,which results in a discrepancy between the data distribution and the binary quantization criterion.Therefore,it is necessary to design a new regularization method more suitable for binary quantization to move the parameters towards ±1.We also use a variant cosine variable for the tail data of the weights to improve the accuracy of binary quantization,considering that some weights are far from the zero point and cannot be corrected well by W regularization.Consequently,we achieve 93.68% accuracy with the quantification of the ReActNet-18 network on CIFAR-10 and 92.68% without batch normalization equivalence,which effectively guides the optimization of parameters in different locations and improves the convergence speed and accuracy.(2)We propose a binary quantization method for deep neural networks based on feature distribution adjustment.To address the problem of information loss caused by dispersion of binary network accuracy parameters and the disappearance of semantic information caused by improper distribution optimization,we guide binarization through feature distribution adjustment to optimize the parameter distribution further and reduce the information loss caused by dispersion.We adjust the parameter mean-variance distribution and balance the parameter distribution by designing grouped excitation module and feature refinement module.We also adjust and optimize the threshold segmentation position to balance the binarization activation effect and to maximize the retention of semantic information.The final quantization using ReActNet-18 network on CIFAR-10 resulted in a loss of only 0.40% of network accuracy,which is higher than the current mainstream advanced binary quantization algorithms.Better results are achieved in different backbone networks and using different datasets,effectively preserving network information.(3)We propose a binary quantization method based on the enhancement of deep and shallow feature information.As the binary neural network mainly focuses on extracting deep layer features,we extract the shallow layer information(image color,shape,texture,etc.)by adding penalties for detail features.We use a semantic information regularization module to avoid the inappropriate preservation of shallow information caused by overfitting the activation function,making the deep features more discriminant.The accuracy quantification of the ReActNet-18 network on CIFAR-10 achieved 93.03% while 93.05% on VGG-Small.Our method can fully utilize the network feature information and perform steadily on other networks and datasets.