Hierarchical Sliding Windows-based RNN For Activity Recognition

Long Short-term Memory(LSTM)has dominated the machine translation tasks and been applied to the whole area of Natural Language Processing(NLP),after Google's Neural Machine Translation(GNMT)published by Google in 2016,which is based on LSTM,where translation errors were reduced by more than 60% compared to Phrase-based Machine Translation(PBMT).Now,LSTM has nearly been applied to every area involved time processing,such as finance,economy,predicting the preference of customers and autonomous vehicles.However,the long-range dependency problem limits the length of any Recurrent Neural Network(RNN)models including LSTM,which means that many of time processing tasks are unsolvable challenges.Since the lengths of videos as sequences on activit y recognition tasks are very long,we believe that it is a good decision to solve long-range dependency problem on activity recognition tasks which are valuable to be researched.This paper proposes a hierarchical architecture named Hierarchical Sliding Windows-based RNN(HSWR)consisting of RNNs which are windows sliding on sequences for compressing information and reducing the lengths of sequences level by level.However,the original HSWR is not convergent during training,for solving the convergence problem,we design Second Half Accumulation model(SHAM)which adds the last half of a sequence where elements are output vectors of RNN on one RNN window.We design an accumulation model inspired by Attention Mechanism-based LSTM and Residual Network,which accumulates all elements in a sequence on one RNN window,is still failed to be convergent during training.Considering that the difference between SHAM and the failed accumulation model is the ratio of the number of elements added on one window and the length of the window,we design Gap Accumulation model(GAM),where elements are added every second element,to confirm that the key of convergence is the number of the elements added instead of the position.A simple database consisting of two sequences whose lengths are modifiable is generated to evaluate the ability of models on solving long-rang dependency problem and where elements are reals but not vectors for quickly calculating and to verify whether models are convergent.We combine CNN with RNN based on Long-term Recurrent Convolutional Networks for activity recognition tasks,and evaluate our models on KTH and UCF101 databases.