The Design And Implementation Of Interface A Signaling Data Collecting System Based On Apache Flume

The rapid development of mobile communication industry brought mobile terminal number of exponential growth,and mobile terminal in communication system interaction process of signaling data also showed blowout type growth,how to collect such a wealth of signaling resources and explore the commercial value become a new topic.In GSM,No.7 signaling carries a lot from switch the device on the basis of the data information,through the acquisition from a interface signaling data of GSM network can be of different business process in the original signal data for data extraction,in view of the demand for such,need a set of complete a interface signaling collection system for signaling data collection,extraction and persistence function is complete,and then for the next step business information mining provides the foundation of powerful data mart.In this paper,the design of the system it is the demand as the starting point,combined with the technology of data collecting and processing,based on Apache flume for secondary development.The system can smooth docking of all kinds of signal collecting equipment and efficient implementation of signaling data collection,cleaning,to for a long time and real time stream output.At the same time,the distributed coordination function is realized based on Zookeeper,which ensures the high availability of the system in the process of processing massive data.In addition,the system on the upper application layer(data mining)exposed service interface,to for external application of initiating the query request and response results;and the lower signaling data gathering,processing nodes of the cluster load state,the abnormal log statistics,thus to system administrators to provide a friendly management interface.This paper first analyzes the current problems existing in the signal acquisition system,so as to extract the new design of the signaling acquisition system functionality and performance requirements,and analyzes the necessity and feasibility of system development based on Flume;after the key extracted according to the needs of all kinds of data in the system are analyzed,the establishment of the model and the data storage scheme,the signaling data gathering,cleaning,conversion,data persistence,data output,data management system,the ability to coordinate various processes;at the same time the docking multiple signaling data source,to solve the system output at the same time to various types of storage location,cluster distributed coordination key to ensure high availability,data open interface gives the solution,complete the system of the total Body design and detailed design;the last unit testing,integration testing,performance testing also to ensure the stability of the system.