Optimization For The Binarized Deep Neural Networks

Deep neural networks are well known to achieve outstanding results in many domains.However,most high-performance deep neural networks are associated with highly complex net-work structures with a large number of parameters,which restricts their deployment,especially in embedded devices.It is widely acknowledged that typical deep neural networks are associated with high redundancy.Therefore,how to reduce such redundancy thereby decrease the computa-tional and space complexity of deep neural networks without significantly lowering performance is an important research problem.To overcome these obstacles,many approaches have been pro-posed,and the binarized neural networks is one of the important research directions,which maps the weights and activations to+1 or-1.Neural network binarization reduce both memory us-age and computing cost drastically,but it is often associated with reduced expressive power and generalization ability.In this paper,we propose a series of strategies to improve the perfromance of BNNs.Firstly,we propose to insert a scaling layer before the Softmax nonlinearity to overcome the learning difficulty associated with large logit scale.To further improve the recognition accuracy while maintaining efficiency of a BNN,a special new neural network architecture is proposed.Finally,we further improve network performance through network distillationWith those proposed strategies,we achieved better performance than previously published results of binarized neural networks on CIFAR-10 and ImageNet.