Research On Spark Based Merging Iteration And Its Application On Financial Analysis

With the coming of information era and the rapid development of fields like the Internet,sensors and bioinformatics,data has grown explosively.And with the development of fields like data mining and machine learning,more and more iterative algorithms have been applied in many fields,such as social network analysis,high-performance computing and recommended system.Meanwhile,the data analysis algorithm is desired to compute in the whole data rather than local data,but the whole data is relative.So the data analysis algorithm compute in both whole data and local data.After the local data merges to the whole data,the result of the local data were pooled and the result of the whole data need recompute.So how to get the final result by the iteration result of the local data,can greatly improve the efficiency of iterative computation in financial analysis.In this paper,we proposed a merging iterative computing model which was suitable for the majority of the algorithms.Without sacrificing the iteration accuracy,this model can obtain new iteration data by means of the iteration result of the local data.Mergeing iterative computing model includes the partition iteration step,loss assessment step and iterative merging step,the partition iteration step was finished before the merging iteration started.By means of data validation and analysis,this paper proved that the merging iterative computing model was correct and advantageous.In the implementation part,this paper improves and realizes the merging iterative computing model based on Spark framework.Finally,by means of a large number of experiments,this paper testified the correctness and the advantages of the proposed merging iterative computing model that achieving the goal of speeding up the efficiency of financial analysis algorithm.Therefore,the merging iterative computing model has a guiding significance in the field of financial analysis.