Research And Implementation Of Economic Dynamic Management System Based On Spark Technology

The emergence of big data has promoted the development of science and technology,which has provided a new opportunity for the financial system,at the same time,that also made the application system have higher requirements in data storage and analysis.Therefore,it is urgent to study related storage and analysis and processing technologies of big data.In this context,Hadoop,Spark and other big data technologies have gradually emerged.This paper mainly studies how to integrate Saprk and other big data technologies into the traditional enterprise financial management system,and finally constructs a new type of financial management system oriented to users,management and analysis,so as to solve many problems such as the current enterprise financial data redundancy and clutter,the quantity and multiple sources collection,the poor data sharing ability,the slow data query and the analysis difficulty.In addition,the operation of large table connection query in the big data processing module of the economic-dynamic management system built in this paper is quite frequent.Howerver in Spark,due to the large data witch does not conform to the join condition involved in the Shuffle term,leading to frequent I/O operations,which makes the Join operation performance low,so this paper will study and optimize the equivalence connection of Spark large table.In this paper,specific research route is divided into three steps,first of all,building economic dynamic management system with depended on the mature Spring MVC framework and Web technology,to complete the traditional,small amount of data of relational data processing analysis needs.Then,Large data technology such as Hadoop and Spark are added to complete that requirement of batch data processing and real-time data analysis in the economic dynamic management system.Finally,a method of using Bloom Filter to pre-filter most data that does not meet the connection conditions to reduce the data entering the Shuffle phase is proposed to optimize the Join operator provided by Spark and improve the connection efficiency.