The Design And Implementation Of Mobile Users' Behavior Data Query And Analysis System Based On Spark

In the information society,mastering the information is the core competitiveness.At present,the use of mobile devices to complete daily tasks has become the new normal,and the resulting massive mobile user behavior data has huge social value behind it.It is of great practical significance to process and analyze mobile user behavior data quickly and mine mobile user behavioral characteristics,laws and anomalies effectively and deeply.This system is a one-stop big data query and analysis platform for cleaning,querying,and analyzing collected mobile user behavior data.The system design is based on the Spark platform and the HDFS distributed file system.It provides data processing and analysis services in the form of Web calls and Open API calls and makes the Spark platform Web-based.The system imports different types of mobile user behavior data into the HDFS distributed file system,and cleans the data according to the characteristics of various types of data.According to the fields of different types of data,the data query conditions are generated dynamically,and the SQL statements are automatically generated based on the selected query conditions.Then,the Spark SQL,a distributed SQL query engine,is called to parse and execute the query on the distributed data warehouse.Using the most representative data of mobile user behavior data,CDR data,according to the user's daily behavior patterns,we extract features and design analysis models.We analyze the behavior of mobile users from call records,moving trajectories and social perspectives,and mainly extract the semantic trajectory features of CDR data.Semantic behavior patterns are mined separately for semantic trajectory sets at each time segment,and the maximal semantic behavior pattern similarity algorithm is used to calculate the semantic similarity between each two patterns.It is used to measure the similarity of the behavior of each time period,considering that the behavior of the time period with low similarity is suspected to be abnormal.Through functional tests and algorithm experiments,the effectiveness and feasibility of the system are verified,and the prediction also reaches a certain effect.The system shields the details of Spark platform and enables non-technical users to process and analyze big data through Web pages.It can assist users in making decisions and perform semi-automated or automated abnormality prediction.