Improved Long Short-Term Memory Base On Continuous Skip Mechanism

With development of Deep Learning,Long Short-Term Memory(LSTM)are widely used in various industries.It performs well in different tasks,especially when we work with sequences.Its excellent inference ability comes from its three inner gates: input gate,output gate and forget gate.However,because of them,LSTM is prone to high computational cost.Today,LSTM is used gradually in more small devices including smart phone and laptop,whose computing ability is limited by its volume and energy.Therefore,LSTM is expected to reduce its computation while the training.This work focuses on how to reduce the computation of LSTM with accepted inference ability,and the main contributions as follow:1.Inspired by the continuous movement of human eyes and distribution of information in objects,we design a new recurrent model using LSTM from two aspects: the first point is how to update or skip a whole hidden layer;the second point is how to update or skip a LSTM neuron on certain time step.This new model is named Continuous Skip LSTM or CSLSTM,which can skip hidden state updates more continuously.2.Equipped by skip gate,the new model needs a new loss function to limit or encourage how many CSLSTM use during the training.Therefore,we design a new loss function used in the training of our model from two aspect: inference ability and computation costed.And there is an intermediate coefficient between them to adjust the ratio of the two,which can be changed by different users who have different demands of accuracy or float operating numbers.Three different experiments have been conducted to demonstrate the feasibility and efficiency of the proposed CSLSTM,which is evaluated by four metrics and compared to seven relevant models.The results have shown a significant improvement in efficiency by the proposed continuous skips while the performance of LSTM has been retained,which is promising for efficient training of LSTM over long sequences.