Research On The Key Technology Of Real-time Computing Of Big Data Supporting The Back Calculation

Real-time calculation of big data is to solve large amounts of data,which needs to be calculated in real time according to the specified business logic,and the calculation results can be stored in real time and can be invoked by external systems.In the production application,the business logic is not a layer unchanged,may needs to be adjusted according to the specific situation.In some scenarios,the historical data that has participated in the calculation needs to be recalculated.In fact,the data that has participated in the previous calculation will not be regenerated from the original data source.This case involves the topic of real-time back calculations for big data,and this is the most important topic to be discussed in this thesis.This thesis takes the scene of big data real-time calculation that needs to be recalculated historical data as the background,research how to solve the problem of real-time calculation of big data,design a method to solve the problem of real-time computation which need to be back calculation.The main goal is to be able to quickly back compute the historical data when the business logic changes in a real-time computing scenario.This thesis learns from valuable experience of previous researches,and combines with the current industry,select a relatively common business scenario as research object,which can back calculate historical data in real time under complex business scenarios,then implement the method which solves the problem of real-time back calculation of historical data in complex business scenarios.This thesis mainly includes the follow aspects:1.Proposing the classification and processing strategy of real-time back calculation scenario: the concept of back calculation is defined,when to do back calculation(the time of the back calculation),and how to do it(the strategy of back calculation)are defined.2.Proposing a processing method of data compensation mechanism when the data source changes: data sources are generated by business services,there may be deletions and modifications according to the services primary key,not just new additions.Aiming at such problems,this thesis proposes a method of data compensation mechanism for deleted and modified data and added data are calculated uniformly.3.Developing a mobile APP real-time report platform that supports back calculation: The mobile APP real-time report platform that supports back calculation is able to perform back calculation quickly in the case of changes in business logic and multi-source data association,and the update frequency of real-time indicators meets the business requirements.The test and analysis of the report platform have achieved the expected goals.