Design And Implementation Of Telecom 4G Big Data Platform For Network Optimization Based On Spark

At present,we are in the era of large data,the explosive growth of information makes the traditional technology architecture can not meet the massive data processing needs,large data platform research and development in this environment came into being.Spark is the most active,popular and efficient large data general computing platform in today's large data field.Spark has successfully built an integrated solution for large data processing based on RDD.MapReduce,Streaming,SQL,Machine Learning and other large data calculation model unified to a technical stack,the developer uses a consistent API to operate all the functions in Spark,making Spark acceleration as a large data processing center preferred and the only computing platform.With the gradual maturity of related technologies,many companies are beginning to study and learn spark technology to meet their own business needs and improve product quality.In this context,my company start to carry out large data network optimization platform application development practice in the telecommunications 4G mobile communication network optimization field based on the spark system,the traditional data analysis methods and means can not meet the mobile data large data age data processing And the aging requirements of the analysis.The platform can be compatible with Hadoop cluster,while efficient analysis and processing of massive data,more than 10 times the traditional way to enhance performance.I am mainly responsible for the design and development of data analysis module and data query and analysis module,including the design of data analysis module flow,coding data analysis,tuning data analysis efficiency,design data query and analysis of the table field,design business functions SparkSQL Query,optimize query performance.This paper mainly introduces the construction of two main functional modules of the platform that I use spark technology to assist large data network optimization in the LTE network of mobile communication operator through processing massive MR measurement data.Firstly,the background of platform construction and the technology of platform construction are introduced.Then,the requirements of platform are analyzed from the aspects of functional and non-functional requirements,and the architecture of the system is determined.Thirdly,it introduces the process design and code realization,performance tuning and result output of data analysis and data query analysis in detail.Finally,it introduces the process of system deployment and the performance comparison with the traditional method,which verifies the outstanding performance of Spark on large data processing and proves the validity and practicability of the work of this paper.The platform has now been put into the company's daily large data processing work,the monthly routine processing of a province mobile company 17 cities up to 2.3T(compressed)after the network optimization data.The results show that the platform can run normally and the performance is excellent,can meet the province to complete a province of scale data analysis and index output requirements monthly,with successful achievement of the desired goal.