Deep Networks With Stochastic Depth For Acoustic Modelling

The natural communication of machine and human beings has always been the dream of humankind.In the past 50 years,great progress has been made in the research of speech recognition technology.Especially since 2011,with the improvement of deep learning theory,the drastic increase of computer performance as well as the accumulation of massive voice training data,speech recognition begin to enter commercial age.Compared with traditional networks,the deeper neural network has better ability of model fitting and expression,but it lacks the applications in speech recognition at present.There are two reasons,on the one hand,due to the effect of gradient vanishing phenomenon,the gradient values decay continuously during the process of error back propagation,which make it difficult to update the parameters close to the network input layer.On the other hand,the deeper network has a large number of parameters,which leads to a lot of time in training and testing.This paper concentrates on the research of stochastic depth network algorithm in acoustics modeling.By adopting the residual network and randomly dropping some residual network blocks during the training process,the gradient vanishing is alleviated.And then combined with the model compression algorithm,under the condition of controlling the scale of the model parameters,a deeper speech acoustic model is trained,which ultimately leads to lower word error rates.The experimental results show that compared with the traditional network,deep networks with stochastic depth can improve the recognition rate of the speech system little when the data is limited,but it can significantly improve the recognition rate of the system when the training data is sufficient.At the same time,the application of model compression technology can effectively reduce the parameters and the amount of calculation during operation.The conclusions of this research have important reference to the design of speech acoustic modeling.