Big Data Storage And Processing Framework For Network Optimization

The global deployment of 4G networks has driven the rapid development of mobile data services.Mobile Internet awareness has become a major factor influencing user experience.With the explosive growth of signaling and traffic data in the network,the traditional voice service-based network quality assessment system as well as the data management and analysis platform can no longer meet the current network optimization needs of data services.Under the current big data technology support,network optimization is a process of system data extraction,collection management and integration analysis.Among them,finding the crux of the impact on network quality and other quality problems is the core,so that achieving a better network by targeted end-to-end optimazation.In thesis for master degree,we study end-to-end network optimization for large data storage and processing framework based on the service data and signaling data in the network and focus on multi-source heterogeneous data integration and processing,spatial-temporal data index.Major innovations and work are reflected in the following aspects:First,the distributed data processing framework based on Spark and MapReduce.In this thesis,aiming at the mass and multi-source heterogeneity of operator data,we design and implement the integration of heterogeneous data from multiple sources and complete the integration of this data based on Spark.Based on MapReduce,a data processing framework for different business analysis is designed and implemented.Second,the spatio-temporal data index framework based on HBase.For the operator's spatio-temporal data,a primary key index structure is improved which considering time,space and user dimension,along with Geo-Hash dimension reduction in spatial dimension and time segment in time dimension so as to better support the query of the historical data of the spatio-temporal range.Finally,results show that our design outperforms the traditional design in spatio-temporal range query.Thirdly,for network optimization analysis and application.Applying perception assessment model for massive data services,and calculating the service assessment score,feature extraction and normalization on the data processing framework for different business analysis.Based on which the user and the base station are monitored to recognize which are not went well.Further,we take the Frequent Itemsets mining algorithm for these base station,to position and delimit its reasons.And finally providing a basis for further network optimization.