Research On Large Scale LTE Signaling Data Processing System

In recent years,the number of mobile data services and smart devices has grown rapidly,and the data traffic of mobile devices has been increasing,which poses a higher demand for the stability and speed of data transmission.LTE network can provide users with higher data transmission capacity.Therefore,the scale of LTE network can be rapidly expanded.So the amount of LTE signaling data also grows rapidly.This presents a huge challenge to the storage,processing and analysis of signaling data.The traditional scale up architecture is difficult to meet the requirements of rapid processing and analysis of massive signaling data.The development of big data technology makes it possible to build a distributed computing system through cheap PC,which is a new way to deal with massive data.This approach increases the processing power of the system by scaling out,which provides a new way for the processing of massive LTE signaling data.First,this thesis considers the massive real-time characteristics of LTE signaling data,and designs a system that satisfies collection,aggregation,processing and storage of massive LTE signaling data.The system can collect LTE signaling data in a variety of formats,and put forward the appropriate data distribution and summarization strategy according to the structure of signaling data and use Kafka to summarize the data.Spark is the calculation module,which is utilized to process the data in real-time.The system designs the storage model according to the requirement of fast storage and querying of signaling data,and uses HBase to store data.Then,in this thesis,we analyze the shortcomings of the real-time processing module in the system.When in real-time processing,the system needs to reshuffle the data after read from Kafka,so that the number of partitions can satisfy the parallel requirements of the computational cluster processing.Repartition needs to shuffle all data over the network,which will result in a large amount of network bandwidth consumption and disk IO consumption.For the consumption of repartition,this thesis proposes a Pre-partition strategy for reading data.LTE signaling data is distributed evenly among multiple Kafka partitions according to the device ID of data when the data is aggregated.At the time of data reading,Pre-partition strategy calculates the partition ID in the computational cluster to which each of the data in the Kafka data source is mapped,and Pre-partition data according to this ID.In this method,the number of partitions can be specified after the data is read to the computational cluster,which can avoid the overhead of repartition.Then in this thesis,an improved data processing rate limit control algorithm is proposed to solve the problem that the number of input data and the computing ability of the cluster is not matched in real-time processing of LTE signaling data.The algorithm calculates the rate limit of the current task processing in real-time by collecting the statistical information of each stage in the real-time processing of the data,and submits the task according to the rate limit.Through this algorithm,the amount of data read by the cluster is matched with the real-time processing performance of the cluster,so that the data can be processed in time.Finally,this thesis verifies the feasibility of the LTE signaling data processing system,and performs experiments to the Pre-partition strategy and rate limit control algorithm in this system.The results demonstrate that the Pre-partition strategy can significantly reduce the processing time of LTE signaling data in real-time read,partition and processing.When dealing with constant data streams and rate-varying data streams,the improved rate control algorithm can reduce the scheduling delay of the task and improve the stability of the system.