Design And Implementation Of Telecom Operator's Income Apportionment System Based On Spark Platform

With the rise of mobile Internet,the number of mobile users has been increased greatly.Each telecom operator has more than 100 million users.As the core business of telecom operator,billing and settlement need to deal with data in PB level.There is a important link of billing and settlement,income apportionment,which dividing the billing revenue into the departments according to the rules and is a processing batch with large data volume,low fault tolerance and high-speed execution.Operators are based on the business database Hana and the MPP database Greenplum to build the income apportionment system.However,the cost of Hana is very expensive,and there is a problem of poor scalability and low stability in Greenplum.How to build a income apportionment system that ensures fast computation and low cost,high scalability and high stability has become an urgent problem for telecom operators.The practical application scenario of this paper is the income apportionment business of China Unicom software lab.Based on the user accounting data of Unicom,the business content is used to calculate the income of each sector according to the given revenue apportionment rules.The business process can be coarse-grained into the following steps:first,data is extracted from the billing system through the data extraction tool to the income apportionment cluster,and loaded in the data warehouse.After the data are all in place,the stored procedure SQLs are executed sequentially in order,and the execution result is written back to the data warehouse after SQL is executed.After the completion of the storage processes of all the provinces,the apportionment will be completed.In this paper,the income apportionment system which divided into data preprocessing layer,computing layer and monitoring layer is implemented with distributed storage and computing technology,and the help with Spark computing engine,Alluxio memory distributed file system,Hive data warehouse,HDFS distributed file system,MySQL database,Python language and Scala language.This paper introduces the practical application background of the system development,and briefly analyzes the technology of Spark computing engine and Alluxio memory distributed file system technology.Then,the system architecture is briefly introduced.Finally,this paper expounds the design and implementation process of the income apportionment system of the telecom operator,which is the focus of this article.The implementation of the new system based on the Spark platform has greatly facilitated the work of the operators,and the system can run steadily without the collapse of the old system.The execution time of stored procedure kinds with large amount of data is improved,and the execution time of the collection process in Henan and Shandong is shortened from more than 1 hours to less than 40 minutes,and the efficiency of the storage process has been improved.The new system has passed functional and non functional tests and is waiting for deployment.