Sparse Optimization And Network Compression Methods Based On The Second-order Information Of Models

The second-order methods represented by newton method have been applied to solve the sparse learning problem and achieved remarkable performance.Comparing with the first-order methods,the main drawback of this type of methods is the unaccepted consumption to reach the Hessian matrix.One way to solve this problem is to search the approximation of Hessian matrix.This paper proposes a new way to approximate the Hessian matrix.This method constructs an approximation to Hessian matrix which is related to the data features based on the L-Smooth property of linear models.Based on the approximation,we construct a second-order subproblem to approach the original problem.Then we utilize Iterative Hard Thresholding method to solve the second-order subproblem in order to solve the original problem indirectly.Our method can be applied to the linear models which are hard to access the Hessian matrix or have no Hessian matrix.We theoretically analyze the method and apply our method to several classic linear models.The experimental results demonstrate that our method is great in the efficiency and accuracy when compared to other methods.The model compression especially the structural network pruning has received great interest in deep learning.Most of the proposed pruning methods employ the zero-order or first-order information to determine the importance of model channels.We propose a new pruning method which utilizes the second-order information of the network to determine the importance of the channels.We apply the power iteration method to get the max eigenvectors of the Hessian matrix which subject to the model parameters.Then we determine the importance of model channels according to the eigenvectors.We apply the structural pruning method to the classic deep neural networks and the experimental results show that our method is a good alternative to prune the models.Above all,we summarize our work as follows:1.This paper proposed a new second-order optimization method for the optimal solution of linear models in sparse constraint.This method can be applied to the linear models which have complex Hessian matrix or have no Hessian matrix.We analyze theoretically and the experimental results prove the effectiveness of our method.2.This paper proposed a network pruning method which utilizes the second-order information of the models to measure the importance of network channels.The method applies the power method to access the second order information of models and prunes the network channels according to the information.The experiment results demonstrate that our proposed method can decrease the model parameters and speed up the models without affecting the accuracy of models.