Design And Implementation Of Data Processing And Portrait Query System Based On Operator Logs

The rapid development of mobile Internet technology and the popularization of smart terminals have accelerated the arrival of the era of big data,and also brought opportunities and challenges to many industries.Every day,hundreds of millions of users use mobile phones and Internet communication services provided by telecommunications operators,which generate real-time application usage data and location behavior data.These two types of data contain great value and can be used These data are used to construct user portraits and help enterprises understand users more comprehensively.The current problem is that the amount of raw data generated by the user is huge.It is impossible to store all the data on the disk before calculating the user portrait information.At the same time,these two types of data also need to mine portraits related to user preferences according to different data characteristics.label.Therefore,it is necessary to adopt different real-time data collection and extraction methods for these two types of raw data,only store the processed result data on the disk,and then design a suitable portrait label and offline data processing method for the characteristics of multiple result data.Calculate the user's profile information,and finally provide external data services.The research focus of this article is to use the current mainstream and mature big data technology to build a complete big data processing platform,process heterogeneous data from multiple data sources,and clean,convert,extract,store,and calculate data after processing.Convert the original user data into user profile data that can be queried externally.This article first analyzes the structure of user-generated mobile application usage data and location data sources,and designs two different real-time data access,data preprocessing and data extraction schemes,and uses the Flink data processing framework for data processing.With the realization of data persistence.Next,based on the characteristics of these two types of persistent result data,we design user-related application portrait tags and location portrait tags,and then combine the two parts of the behavior portrait tags with the user's static attribute portrait tags to form a total user portrait..The calculation of user portraits is based on the Spark data processing framework,which calculates the data collected on the day and the historical data.Design a reasonable data storage structure according to the characteristics of user portrait data,and use a non-relational database HBase for storage.Then design the external portrait data query service,query the portrait result data in the HBase database in real time,use the Spring Boot framework to implement and add the permission authentication function.Finally,the test work is carried out from two aspects of functional test and nonfunctional test,and test cases and test results are given.Ensure the feasibility and stability of the system.This system provides a complete solution for operators to process and analyze user behavior data,and also provides data support for subsequent data analysis.