Dynamics Of Information Propagation In Deep Dropout Networks

Gradient vanishing and exploding problem is closely related to the depth of net?works.A good choice of the initialization of the parameter can avoid the gradient vanishing and exploding problem efficiently.So far,several opinions and conclusions have been brought out and the most completest one has the conclusion that the be-havior of gradient is controlled by two scales Xi,X2.It comes to conclusion that the gradients in dropout neural network with single input can avoid vanishing and explod-ing problem and information in this kind of network can propagate for a very deep depth.While gradients in dropout neural network with two inputs is sure to suffer from vanishing and exploding problem.Therefore,information in this kind of networks cannot propagate deep also the network is hard to train.Besides,there are also some new viewpoints and explanations.This thesis is concerned with the relationship between gradient and scales Xi,X2-We find under a different assumption,the behavior of gradients is controlled by Xi,X2 in a different way.Through inference and experiments,we find there exists the case that gradients will not vanish or explode in dropout neural network with two inputs which means information can propagate deep in dropout neural network with two inputs.We will provide the derivation process,simulation experiments and real data experiment to verify our conclusion.