Research On Streaming Processing Framework And Key Technology For Big Data Of High Frequency Securities

The popularity of information technology in all walks of life,to promote large-scale data generated in different areas,to large data processing has brought new technical challenges.High-frequency securities trading data is a typical "streaming large data",with large-scale data,complex structure,fast flow and so on.How to use the limited system resources to construct a stable,reliable and efficient data processing framework,and complete the data response in the high frequency push flow data cycle is the urgent problem to be solved.Based on the analysis and research of the large data stream processing model,this paper constructs a large data stream processing framework for high frequency securities based on a variety of large data processing techniques,and studies and improves the key technologies involved in the research and improvement Real-time analysis of the scene,to achieve a highly efficient data stream processing,management and query.This paper focuses on the construction of the flow data processing framework which fits the high data characteristics of high frequency securities and deeply studies the key technologies involved in the framework.The main work of this paper is as follows:1.Analysis and design of high-frequency securities for large data flow processing framework.With the Storm flow processing framework and the Redis memory database as the technical prototype,the two are combined and improved,and the streaming processing framework for high frequency securities and the hierarchical processing model of streaming data are designed.2.In view of the insufficiency and shortcomings of the Storm components in the framework,Storm is optimized from the physical,logical and application levels to enhance the real-time processing capability of high-frequency streaming large data.3.Designed to achieve a large memory based on data access to Reds based on the shared memory center.Through the improvement of the Redis memory database,it not only retains the flexibility of storage requirements and scalability advantages,but also consider the efficiency of data I/O,make up the flow processing framework Storm components can not save the state data defects for the upper application Depth mining provides efficient I/O protection.4.The Application of the Framework in the Real-time Analysis of High Frequency Securities.Completed the application of high-frequency securities for large data flow processing framework for the follow-up securities trading strategy development and implementation to provide framework support.