An Empirical Research On Financial Trading Algorithm Based On Deep Reinforcement Learning

Deep reinforcement learning is a new subject in the field of machine learning.It combines the perception ability of deep learning and the decision-making ability of reinforcement learning.Financial trading algorithm based on deep reinforcement learning methods can learn the mapping from the state representations of financial market to the trading decision directly from the market data.Compared with the establishment of the traditional trading algorithm,it does not rely on complex expert experience.Compared with the supervised learning method,it does not need to make an explicit prediction of the market trend,and can output trading strategies directly.The financial trading scenario can be mainly divided into single-asset trading task and multi-asset financial asset allocation task.However,in both of the two tasks,the existing work based on reinforcement learning algorithm exists some problems,such as insufficient representation of market state in Markov decision process(MDP)model,poor exploration of the exploration strategy in the algorithm,and no risk control is introduced in algorithm training which leads to high volatility and large retracement of the algorithm yield curve in the backtesting.In order to solve the above problems,this thesis proposes the corresponding MDP models and the trading algorithms based on deep reinforcement learning methods in both the single-asset trading task and financial asset allocation task.In the single-asset trading task,this thesis makes use of various features to exploit the market state,making the state representation closer to the true market state,and proposes a reward function that considers the asset retracement,which takes use of the rewards obtained from the environment to guide the algorithm to reduce the withdrawal of revenue during the training process,as well as the risk of the strategy.Based on the DQN algorithm in deep reinforcement learning,this thesis proposes a single-asset trading algorithm,which adds Gaussian noise to the fully connection layer of the deep neural network to drive exploration.It has a better exploration effect than the ?-greedy exploration strategy used in DQN algorithm.The effects of noise in the network are controlled by a set of parameters that are learned by gradient descent like other parameters in the network.Also,with the added noise,the update mode of the target network in DQN algorithm is improved as target value become stable.In the financial asset allocation task,this thesis makes use of technical indexes to exploit the market state and proposes a financial asset allocation algorithm based on DDPG algorithm in deep reinforcement learning,which introduces the entropy of asset weights as a regular term in DDPG algorithm and encourages the algorithm to spread the weights when generating asset weights,instead of focusing on single or several assets,thus reducing the risk of the portfolio.On the other hand,entropy can also be viewed as a supplement to the exploration mechanism in DDPG algorithm to guide the agent for exploration.At the same time,this thesis introduces the prioritized experience replay mechanism to DDPG algorithm,which samples transactions with large TD error with higher probability,thus improving the learning effect of the algorithm.Finally,this thesis backtests DQN-based single-asset trading algorithm and DDPG-based financial asset allocation algorithm in different market environments.In the income assessment,the annualized rate of return of both the two algorithms exceeds other strategies.DDPG-based financial asset allocation algorithm risk assessment has also made significant progress compared to DDPG algorithm.