Autonomous Trading Agent With Reinforcement Learning

This dissertation examines the use of reinforcement learning in autonomous agents that can interact intelligently with financial markets.Stock market trading is used to evaluate and develop a number of machine learning approaches specifically able to handle the challenging characteristics of the financial market trading problem,particularly reinforcement learning.The prediction of change in the stock market is a very difficult task because the underlying patterns that drive market behavior are non-stationary that means useful predictive patterns learned in the past may not be suitable to be applied in the future.Reinforcement learning has not been widely applied in this application domain and the paradigm of reinforcement learning provides a way to allow agents to directly learn trading decision models with more degrees of freedom than many other techniques,for example without a requirement to preset particular thresholds that define certain signals for buy or sell decisions.The change of price can naturally be viewed as a reward and this will avoid the drawbacks of labeling data related to setting thresholds if the problem is formulated as a supervised learning problem.Reinforcement learning can also avoid costs needed for labelling of examples and constructing a training data set.However,in a study of the literature,we find that existing research applying reinforcement learning algorithms to generate trading decisions does not in general account for the environment being non-stationary.The approaches described in the previous literature describe applications of a single agent that may not be recalibrated and learning methodologies that sometimes can be susceptible to limitations from being stuck in local optima.The proposed methods in this dissertation mitigate some of these issues by using multiple agents and a multi-stage learning model where the agents compete to recommend the best decisions.Our approach combines online learning with reinforcement learning.Online learning is used to select a recommendation from a set of agents at the decision point in real time;in addition,the technique is able to relearn and adapt the set of decision models based on recent data.To develop the approach with reinforcement learning,this research produced new methods that can modify the process of training reinforcement learning agents to give additional focus to recent data.The novel methods are evaluated with empirical analysis using data from a range of international and Chinese stock markets.We find that agents based on the proposed methodology are able to outperform other machine learning methods in terms of various metrics and including application specific measures of risk and return that are accepted in the finance industry.Experiments show that agents which use online learning and reinforcement learning achieve higher return over a benchmark trading method buy and hold and using online learning provides substantial improvement in performance of a Deep Q-learning agent.Notably,during the financial crisis,the On-Line/Reinforcement Learning(OLR)agents can stay profitable many cases while other agents suffer a loss in all tests during this time.