Research And Implementation For Distributed System For Financial Quantitative Analysis

In the financial securities investment industry,quantitative analysis refers to an analysis method that quantitatively analyzes historical securities data in accordance with quantitative strategy logic and eventually converts them into valuable trading signals.The quantitative analysis application first needs to load the historical securities data from the underlying data source,then process the data with the user-defined quantitative strategy logic,and finally return the acquired signal data to the upper application.With the development of big data technology,the demand for financial quantitative analysis of large-scale historical securities data is increasing.However,the traditional financial quantitative analysis system is based on the single machine memory,does not have the ability to process large-scale historical securities data.In addition,the big data systems have a high learning and programming threshold and are not easy to use.It is difficult and inefficient for quantitative analysts to design and implement the big data programming model and interface for quantitative analysis.Aiming at the above problems,this paper proposes a distributed financial quantitative analysis programming model and framework,and designs a distributed financial quantitative analysis system based on the model and framework to support the quantitative analysis application of large-scale securities data.The main work and contributions of this paper are as follows:(1)This paper proposes a distributed financial quantitative analysis programming model and framework based on the pipeline model,and designs a set of distributed financial quantitative analysis programming interfaces based on this.(2)Based on the above programming model and framework,a distributed quantitative analysis system called Alchemy is designed and implemented.Alchemy builds a multi-dimensional index for the quantitative analysis job to perform fine-grained parallel computation.Alchemy also provides distributed data storage,parallel computing and other services for quantitative analysts.(3)In order to further improve the reliability and performance of the financial quantitative analysis system,the system fault tolerant and security mechanism is designed and implemented,and completed three underlying optimization methods,including source data cache optimization,metadata index optimization based on random sampling and distributed file merging optimization based on Data Frame index.(4)The experiments verify the effectiveness of the distributed financial quantitative analysis system and three underlying optimization methods.Compared with the RQAlpha,Alchemy can achieve up to 3.91 x and 4.02 x speedup in the two dimensions of stock and trading date.The performance improvement of the quantitative analysis job is 42.67%,27.63%,17.37% in the data load stage,the parallel computing stage and the file merging stage.(5)As a case study,Alchemy has been deployed in the production environment of Huatai Securities,and compared with original financial quantitative analysis system of Huatai Securities.The experiment results show that Alchemy can achieve up to 100 x and 336 x speedup,with a near-linear acceleration ratio.