Stochastic Deep Learning Based Time Series Forecasting Techniques

Time series forecasting is of great importance for a learning system in dynamic environments,playing a vital role in many real-world applications,such as energy,traffic,finance,and industry.Recent studies have shown that deep learning technique has shown intriguing prediction performance,leading to extensive research on the applications of the DNN models for time series forecasting.However,represented by the convolutional neural network and recurrent neural network models,the deep neural network–based forecasting models have complex architecture-related parameters and rely on gradient-based algorithms to train the weight-related parameters,making it extremely time-consuming and challenging to well establish a forecasting model.As an alternative method of training the neural network model,the stochastic mechanism fixes the weights of the input layer and the hidden layer after random initialization,and uses a simple and direct closed-form solution algorithm to calculate the weight of the output layer of the model,which can effectively improve the construction efficiency of the neural network model.Therefore,this study investigates time series deep learning prediction modeling technology based on the stochastic mechanism,pays attention to the structure selection problems under the specific structures,and proposes the hidden structure construction method of the stochastic convolutional neural network and the output structure selection method of the stochastic recurrent neural network.On this basis,the optimization problems under the universal structure are explored,and the feature selection method as well as the parameter optimization method of the stochastic deep neural network prediction model are proposed to further improve the prediction performance of the models.At the same time,the application research is carried out in combination with important application scenarios,such as influenza prediction,crude oil price prediction,electricity load prediction,electricity price prediction,and so on.The main contributions of this dissertation are summarized as follows:Focusing on the model construction problem of stochastic convolutional neural network–based forecasting model,a novel error-feedback stochastic modeling strategy and greedybased selection algorithm are proposed to craft the random convolutional neural network for time series forecasting.The proposed method suggests that random filters and neurons of the error-feedback fully connected layer are incrementally added to steadily compensate for the prediction error during the construction process,and then a greedy-based filter selection is introduced to enable the model to extract the different sizes of temporal features.Comprehensive experiments on the simulated dataset and several real-world datasets show that the proposed method exhibits stronger predictive power and lower computing overhead compared to trained state-of-the-art deep neural network models.Focusing on the model construction problem of stochastic recurrent neural network–based forecasting model,a novel state mask strategy with particle swarm optimization is proposed to construct the random recurrent output structure for time series forecasting.Based on the investigation of the stochastic implementation of different recurrent output structures of training-based recurrent neural networks,the proposed method adds a mask to each step of the recurrent hidden features,and then a particle swarm optimization based mask selection is introduced to evolve the mapping relationships from recurrent hidden features to their targets,which improves the learning ability of the stochastic recurrent output architecture.Compared with the stochastic recurrent neural networks with the existing output architecture selection method,comprehensive experiments on the simulated dataset,electricity load dataset,and outside temperature dataset demonstrate the superiority of the proposed method.Focusing on the input feature selection problem of stochastic deep neural network–based forecasting model,a novel dual feature-structured selection method with tree-structured parzen estimator is proposed.Based on the ability of deep neural architecture that can model multiple dimensions in each input time step,a multiple-step-dimension two-dimensional feature structure is established with a moving window schema.The proposed method adds a mask to each input step to represent the two-dimensional feature structure with the combination of step dimension and step mask,and then tree-structured parzen estimator is introduced to evolve the feature structure,which improves the learning ability of the stochastic deep neural networks.Compared with the stochastic deep neural networks with the existing feature selection method,comprehensive experiments on the simulated dataset,electricity load dataset,and electricity price dataset demonstrate the superiority of the proposed method.Focusing on the model construction problem of stochastic deep neural network–based forecasting model,a novel error-feedback triple-phase optimization strategy is proposed to grow stochastic deep neural network–based predictor with mixed deep neural architectures.The proposed method incrementally adds diverse deep subnetworks to the network,where the output weights of the subnetworks are calculated via ridge regression to improve the robustness of the constructed model,and the parameters of the subnetworks are evolved with pre-tuning,sub-tuning,and reg-tuning optimization,making the network take advantage of different deep neural architectures.Compared with the existing stochastic deep neural networks,comprehensive experiments on the simulated dataset,air pollution datasets and electricity load datasets demonstrate the superiority of the proposed method.