Research On New Deep Reservoir Computing Models

Reservoir Computing(RC)is a simple and efficient framework for training recurrent neural networks(RNNs).RC based models have been widely used in time series predictions,dynamic system identification,etc.However,compared with deep learning models,RC networks has its own limitation on the shallow structure of the whole RC system(only single reservoir),and the learning of the only trainable weights in output layer depends on simple regression techniques.In this way,RC methods have not been used in other challenging fields such as action recognition.Based on a classical RC model called Echo State Network(ESN),this paper explores the possibility of integrating the ideas and techniques of two main-streamed frameworks: reservoir computing and deep learning(DL).Two novel deep reservoir computing models are proposed in this paper.Specifically,the main content of this work includes two aspects:1)A novel hierarchical RC model with multiple projection-encoding called Deep Reservoir Network(DRN)proposed in this work.A DRN stacks multiple reservoirs and unsupervised encoders alternatively in a pipeline way.There are connections between reservoirs of each layer and outputs,and these connections can be trained by a simple regression technique.The main advantages of DRN are that it not only remains the computational efficiency of traditional RC networks(e.g.,it didn’t depend on the process of back-propagated through time),but also can abstract multi-scale dynamics feature via unsupervised encoders between neighbor reserovirs,which is learned from DL clues,such as stacked RBM.The experimental results show that this multiple projection-encoding-based DRN outperforms the existing hierarchical RC models in modeling time series data.At the same time,by visualizing the system’s dynamics with optimized settings under specify prediction task,the results show that DRNs can capture much more rich multiscale dynamics at different layers,while other baseline models do not show dynamics as much rich as DRN.2)Another work in this paper is Convolutional Echo State Network(ConvESN),which one is based on encoding by reservoir and decoding by convolutional technique.This proposed ConvESN is suitable for general tasks of time series classification.Its motivation is that the random and large reserovir can produce a high-dimentional dynamical features,but these features can’t be fully understood by simple regression equation in standard RC framework.In this way,using some strong-decoding technique like convolutional filters can largely help to capture complex dynamics in reservoir.Furthermore,in order to model time series with structural information such as human skeleton-based data,two extended ConvESNs with two multi-step channel fusion strategies are proposed(they are ConvESN-MSSC and ConvESN-MSMC)and evaluated on well-known UCR time series repository and four challenging action recognition benchmarks.The results demonstrate the effectiveness of the proposed ConvESN on all tasks.