Understanding ResNet From An Iterative System Perspective

Convolutional neural networks,the most popular learning tool for computer vision in recent years,are based on the basic principle that features are selected and extracted through convolutional kernels.The extracted features are used to represent the image,then one can carry out classification and recognition.However,due to the introduction of non-linear operations in the network framework and the difference in network depth as well width,it is difficult to carry out in-depth theoretical analysis of learning networks.Classification is a major research direction in computer vision,so research on the underlying principles of deep learning classification can further optimise the network modules and reduce the redundancy of training.It also provides the basis and theoretical guidance for constructing new efficient networks.This paper expounds the Residual Network from the perspective of an iterative system,which is the most classical classification learning framework.Before the emergence of ResNet,the network was prone to degradation in the process of learning due to increasing depth,resulting in fewer features that are useful for classification and thus affecting the classification results.ResNet addresses this phenomenon by cleverly adding skip connections.The classification accuracy and stability are greatly improved with deeper depth,and useful features do not disappear.This paper introduces the viewpoint of an iterative system to understand ResNet,The overall module of convolutional kernels and jump connections in the ResNet network can be viewed as an iteration matrix,and then the classification stability of the ResNet network is investigated by introducing the condition number of the matrix.To illustrate the greater stability of the ResNet network,this paper also introduces a basic network with convolutional layers,non-linear units,downsampling,and a fully connected layer in the model.It differs from the ResNet only in the presence or absence of jump connections,and we compare the theoretical results of the stability analysis of the base network and the ResNet in detail.Results show that the condition number of ResNet module is smaller than the condition number of the base network,this conclusion can explain that the addition of jump connections in the network module makes the ResNet model itself more stable than the base network.Meanwhile,the classification accuracy of the network is visually described by the median principal angle,then the corresponding theoretical analysis is done.Finally,the validity of our theory is illustrated on the Dogs vs.Cats dataset,the Kaggle: Animals10 dataset,and the Image Net 2012 dataset respectively.The main innovative work in this paper is as follows.1.For the classification stability of ResNet,we use random Gaussian weights as the initialized convolutional kernel of the network,and introduce the principle that the smaller the condition number of the matrix,the better the stability of the model.Based on the convolutional action process as a linear transformation,the principle explains the theoretical reasons that ResNet outperform the underlying network in the classification process and network training process.2.ResNet obviously improves the classification accuracy,but most of the studies on its accuracy are based on the overall part.Besides,there is no good representation of the classification effect between and within specific classes.The introduction of median principal angle can well illustrate that ResNet can not only increase the inter-class distance,but also decrease the intra-class distance.Therefore,we calculate the median principal angle between and within classes for different datasets,and collate the experimental results to fully illustrate the trend of angle changes between and within classes.