Research On Partial Information-Based Deen Learning Methods

In recent years,deep learning methods,featured by neural networks,have evolved rapidly and gained great success in numerous fields.However,in order to learn sophisticated target features,deep learning models usually contains large-scale weight matrices,which lead to large storage occupation and high computational complexity.These disadvantages make it hard to complete inference and training of deep learning models on small edge devices and pose challenges to the applications of this technology.To reduce the computational complexity of the deep learning methods,approximate decomposition method of random matrix is introduced for the first time and novel high-performance and low-complexity neural network layer structures with corresponding algorithms are proposed.With only a few calculations on small matrices,the inference and training process of large-scale deep learning model can be completed without affecting its generalization ability,which effectively reduces the temporal and spatial complexity and provides a new method and perspective for the lightweight deep learnings on edge devices.The main innovation of this paper are listed as follows.(1)To solve the problems of high complexity of inference and training processes brought by large-scale weight matrices in the fully connected layers,a new low-rank connected layer structure is proposed by using CUR matrix approximate decomposition method,which broadly decomposes a large-scale weight matrix into three small matrices.The inference and training of lightweight low-rank connected layer can be finished by only calculating and updating the three small matrices.The new structure and method are able to compress the weight matrices by over 10 times while the test accuracy drops less than 1%.(2)To solve the problems of high complexity of inference and training processes brought by large-scale weight matrices in the convolutional layers,random sampling and approximate calculation methods are introduced to decompose the large weight matrix into three small matrices and a new low-rank convolutional layer structure is proposed.By replacing the convolution with a few small matrices'multiplications,the inference and training of lightweight low-rank convolutional layer can be implemented.Without degrading the test performance,the new method is able to compress the weights in convolutional layers by 2?3 times.(3)To solve the problems of high complexity of inference and training processes brought by large-scale input feature matrices in the convolutional layers,a new convolutional layer structure based on input features decomposition is proposed by using three small matrices to approximate large input feature matrix with the help of random matrix approximation methods.By replacing the large-scale matrix multiplication with a few small matrices'multiplications,the calculations of convolutional layer can be implemented with less storage and computational resource required.The sophisticated design lowers the approximation error and is capable to reduce the convolutional computations by 3?5 times.In the end,three new structures and methods are also synthesized to build a low-complexity deep learning model with the advantages of little generalization decay,small storage and low computational complexity.