Quantitative Strategy Backtest Evaluation Based On CSCV

After more than 40 years of development,quantitative investment has gained wide popularity due to its advantages of sustainable excess returns.Nowadays,the development of computer technology in the era of big data is unstoppable.With the continuous iterative updating and optimization of machine learning algorithms,relevant researches in the field of investment must keep pace with the times,and quantitative investment strategies are emerging in an endless stream with each passing day.At the same time,it is the latest advances in machine learning,artificial intelligence and the availability of billions of high-frequency data signals that make model selection a challenging and urgent need.Many investment firms and fund managers rely on backtesting(performance simulations based on historical market data)to select investment strategies and allocate capital.Financial discovery usually involves identifying a phenomenon with a low signal-to-noise ratio,usually by using computational power to calibrate the parameters of an investment strategy to maximize its performance.However,the low signal-to-noise ratio can easily lead to the calibration result being based on the past noise rather than the future signal,thus backtest overfitting occurs.Backtest overfitting is an investment strategy that performs well during backtest,but performs poorly in practice.It is not easy to evaluate the backtest overfitting,and there is a lack of relevant research.Scholars such as David H.Bailey and Marcos L'opez de Prado have paid attention to this issue and conducted relevant researches.In 2017,they proposed a quantitative definition of backtest overfitting,and proposed a method called“combined symmetric cross validation(CSCV)" to estimate the probability of backtest overfitting,and verified the accuracy of CSCV method by Monte Carlo(MC)simulations and applying Extreme Value Theory(EVT).The effectiveness of CSCV method is verified by several test cases under different backtest overfitting degrees(full overfit,high overfit and low overfit).Quantitative stock selection is an important part in the field of quantitative investment.On the basis of literature research,this paper selected the most widely used three algorithms--SVM,RF and XGBoost algorithm,built a stock selection model,and adopted Markowitz's mean-variance method to optimize the portfolio of stock pool,thus forming a complete investment strategy.In this paper,300 constituent stocks of HS300 are selected as the research stock pool,and empirical results show that the three strategies all outperform HS300 index in the out of sample.This paper introduces the definitions of backtest overfitting and the probability of backtest overfitting quantified by David H.Bailey et al,and estimates the probability of backtest overfitting of three quantitative investment strategies based on CSCV method.In the evaluation stage,comprehensive backtest evaluation of investment strategy is carried out from the new perspective of the occurrence level of backtest overfitting on the basis of usual evaluation perspective.This gives an assessment of whether popular strategies that seem to perform well have the ability to generate excess returns in real investments.The study found that although the investment strategy based on "XGBoost-MV" has better performance than the other two strategies during the backtest period,its backtest overfitting probability is as high as 57.1%.It is reasonable to judge that the backtest overfitting occurred in this strategy,which means that the backtest performance of the strategy was not authentic and could not be applied to the actual investment.The return rate,Sharpe ratio and other indicators of the investment strategy based on "SVM-MV" and "RF-MV" are not as good as those of "XGBoost-MV" during the backtest period,but the backtest overfitting probability is 11.9%and 18.2%respectively.Therefore,the investment strategies based on "SVM-MV"and "RF-MV" are both low-risk overfitting strategies,that is to say,the backtest results of these two strategies are more authentic.Based on the authenticity comparative evaluation of different strategies,we can consider to choose the strategy with the best performance under certain overfitting probability threshold rather than the strategy with the best performance under backtest.For strategies with better performance,a more critical perspective is added,which helps researchers to find more realistic strategies.This provides new thinking on strategy choice for real world investors and has important practical significance and application value.At the same time,this paper also provides a new research perspective for the future research.For example,the strategy selection method with higher reliability can be studied.In the case of setting the same performance statistics of the strategy(e.g.,Sharpe ratio),when the performance differences of the performance statistics are within a certain range,we can adopt the strategy with higher backtest reliability.