Research On Time Series Model Based On Reinforcement Learning

Time series data can be seen everywhere in life,such as daily temperature in weather forecast,pork prices posted in supermarkets every day,consumer price index released monthly by the National Bureau of Statistics,etc.People have been studying time series for a long time.The earliest time series analysis can even be traced back to ancient Egypt 7,000 years ago.They recorded the fluctuations of the Nile River day by day to predict floods and guide agricultural production.It is expected that by analyzing time series data,the rules can be found,so that the future situation can be better predicted and the current behavior can be guided.On the other hand,from Go to e-sports games,from robot control to autonomous driving,reinforcement learning has moved from a concept on paper to a reality within reach.As a technology that is expected to realize general artificial intelligence,reinforcement learning carries the researchers’ good wishes for the future,that is,to allow computers to have real learning capabilities,to continuously improve themselves in the interaction with the environment,and to achieve unlimited potential.The real purpose of time series analysis is to guide people’s behavior,and the output of the reinforcement learning model is also the behavior that should be done at the moment.If the time series data is used as the input of the reinforcement learning model,the output is still the action to be done at the moment.An end-to-end time series decision-making system was developed.Such systems have been applied in many fields such as anomaly detection,disease diagnosis,network security,financial trading,etc.,with broad research prospects and great potential.However,there are still many problems to be solved in the research of time series reinforcement learning model,such as data representation of time series and data augmentation of time series.In addition,some tasks also need to use constrained reinforcement learning and imitating expert reinforcement learning due to the limitations of the environment or data set.This paper studies three aspects of the time series reinforcement learning model,the details are as follows:The first aspect is the data representation of time series.In this paper,we propose an approach that combines supervised learning with reinforcement learning to achieve the representation of time series data by sharing network parameters.There are already many network structures for processing time series data in deep learning,and the task of deep learning is often supervised learning,that is,there are labels as "correct answers" in the training data set,and the deep learning model needs to extract useful information from the input.The deep learning model needs to extract useful features from the input,and then obtain the output of the model according to these features.The goal of learning is to make the output as close to the label as possible,and the model’s ability to extract features will become stronger and stronger in this process.Combine reinforcement learning with supervised learning and let them share a part of the network structure,so that the reinforcement learning model can use the data representation ability of the deep learning model to obtain better learning results.The second is data augmentation of time series.This paper proposes a data augmentation method based on fractal theory,and a method to measure the similarity of datasets before and after data augmentation.Reinforcement learning algorithms require a large amount of training data to obtain a satisfactory model.If the amount of data in reality is limited or the cost of acquiring data is too high,data augmentation is needed to increase the amount of data.The method proposed in this paper to measure the similarity of datasets is based on mean,variance and JS divergence.If the two datasets reach a similar threshold under this measurement method,the data augmentation method based on fractal theory can be applied to augment the coarse-grained time series dataset with the fine-grained time series dataset to achieve the effect of data enhancement.The third aspect is the expert strategy reinforcement learning model.This paper proposes an options trading framework based on constrained reinforcement learning.The ideal reinforcement learning model in people’s mind can be trained from random parameters,and a satisfactory model can be learned only by the observed state,the action taken,and the reward of environmental feedback.However,there may be some states in the real environment that should not be explored,such as a serious car accident in an autonomous car.In this case,a constrained reinforcement learning model is required to limit the actions of the agent and avoid reaching certain states as much as possible.The options trading framework proposed in this paper is based on constrained reinforcement learning,in which the protective stop-loss strategy can largely avoid unbearable losses.