Design And Implementation Of Data Service System Based On User Portrait

Analyzing and using user portrait data is of great significance for the refined operation of Internet services.A large amount of user basic information and behavior data contains great business value that can be mined.Users can be labeled in various dimensions based on these basic data,so that user characteristics can be fully displayed.Putting operational activities into a group of precise users can greatly expand profits and reduce costs for corporations.This thesis designs and implements a user profile data service system based on a large amount of operational data generated daily by Internet companies.The system processes offline data and real-time data based on big data technology,and it produces and manages user portrait label data.The main research,design and development contents of this article are as follows:(1)Real-time retrieval service for user portraits module.The system mainly uses data retrieval technology for key/value structure data to query the value of user label attributes,and determines whether a user entity conforms to the crowd rules through logical calculation.(2)Crowd/landmark demarcation service module.This module combines Spark data processing technology and Elasticsearch data retrieval method.By parsing and processing the JSON conditional expressions,it realizes the business logic of defining the set of qualified users based on the tag value combination.In addition,the system platform supports management functions such as crowd creation,sharing,and deletion,and has the same capabilities at the landmark level.(3)Unified label collection service module.By optimizing the label management and approval process,the construction of a standardized label system was completed,which provides metadata information collection and management services for the company,and provides the basis for the data production process at the same time.(4)Data production and monitoring module.In order to obtain standard profile data that meets data service capabilities,the system uses the Hive database and Spark computing engine to store and process offline data in this module.And it combines with the message middleware technology Kafka,real-time computing technology Storm and Tair database Distribute to filter and record data.Through data quality monitoring to realize the alarm of abnormal data production conditions.The server side of the system is based on the Spring MVC and Mybatis development framework.The data processing process is mainly scheduled and completed by Spark jobs,combined with HDFS,Tair,Elasticsearch,Hive and MySQL to complete the distributed storage of data.At present,the system has been put into operation after testing and verification.During this period,it has provided a unified and effective user portrait public data service for multiple business parties of the group.