Design And Implementation Of Reporting System Based On Spark Platform

With the development of 4G services, the size of upgrading business model is growing and business data grows linearly. Real-time reporting system is processed for the whole amount of data by business hall or salesperson,it leads to a period of time to deal double with stress of the database,which dragges down the entire system to be run. For the salesperson, the performance of the audit workd is more obvious. Due to these problems, this paper is designed to cloud-based reporting system.The contribution of this thesis are as follows:The reporting system proposed in this paper mainly contains four aspects,which are data extraction layer, Spark computing layer, business logic layer and interface layer. Data extraction layer’s main work is the real-time data of incremental data extraction,production database uses Ogg(Oracle Golden Gate) synchronized to the mirror database in the process of generating incremental data files,then Kafka message system is used to send the incremental data to Spark computing layer;The role of Spark calculation layer is real-time incremental calculation, using Spark streaming handling subscription Kafka message, parallel processing incremental information,then backuping the increment message to HBase,Redis and HBase,for non real-time calculation,data is obtained from the bhase and calculated by Spark; The role of the business logic layer is to obtain data from the Redis and HBase, the business logic layer is used to the system interface and peripheral interface; The role of the interface layer is to query and display report data, query monitoring information, query statistics system, etc..Finally, experiments are used to verify the advantages of Spark computing, in the case of a large number of data processing, it also proves the effectiveness and practicality of this work.