Adaptive Recursive Residual Networks Based On Implicit Euler

Neural networks have drawn considerable attention in solving various problems such as target recognition and tracking,image reconstruction,and natural language processing over the last decades,working as emerging technical tools.However,the theoretical understanding about neural networks is still lacking.The compositions of the network structure tend to be determined by experiments as well as its effects depend on users’ experience,which limit the development and application of neural networks to a certain extent.Therefore,how to analyze the theoretical properties of neural networks and use them as a guidance to design network structures effectively have also become the research emphasis.From the theoretical perspective of discrete dynamic systems,we focus on the numerical solutions and convergence stability related to ODE(Ordinary Differential Equation),explore the connection between ODE numerical theory and neural network,and use ODE theory as guidance of network design.There are both explicit and implicit schemes in various numerical solution methods about ODE.Explicit schemes have simple manifestations along with solutions when implicit schemes are more complicated but have advantages in numerical stability and other aspects.Therefore,considering parameter convergence and system stability about neural networks,implicit theory is more suitable for the guidance of network design.However,most of existing researches are related to explicit schemes.The connection between implicit theory and network as well as its application are still research problems,which is also the research motivation in this thesis.Specifically,we deduce the implicit scheme(IM-scheme)from implicit Euler by iterations,which is the first network representation about ODE numerical implicit solutions.Moreover,the IM-scheme can be used to explain some existing classics of recursive residual networks.In addition,for the limitations of IM-scheme in balancing model performance and convergence speed,we propose an adaptive implicit scheme(AIM-scheme)on the basis of numerical analysis,also analyze and verify its parameters.In fact,AIM-scheme is an extended model of IM-scheme,allowing the model to have a more flexible convergence interval and better performance theoretically.Aiming at recursive residual networks under IM-scheme,we use AIM-scheme as extension to propose a series of adaptive recursive residual networks for the purpose of improving models and proving implicit theory guidance about network design.Finally,from sparse signal recovery and image super-resolution,we compare the proposed adaptive recursive residual networks with their original algorithms on many kinds of data such as synthetic signal and real images.Under various comparative evaluation indicators,the adaptive recursive residual networks have all significantly surpassed their basic algorithms,which fully proves the broad applicability and effectiveness of the AIM-scheme proposed in this thesis,and further reflects feasibility and efficiency for designing network structure by theory guidance.