The Design And Implementation Of A User Behavior Data Management Platform

With the explosive growth of user data,User Profile can help enterprises understand the real intentions of user actions,then enterprises can make different recommendation,search and advertising for different person.The China Literature has hundreds of millions of users,but there is still insufficient mining of the user's data value.Based on such a background,it is necessary for China Literature to construct a highly flexible,highly scalable and highly concurrent platform that can be used by multiple operators from a large number of users reading action data.This thesis uses Lambda architecture and analyzes the current status of data management methods in the company.By combining the disadvantages of those methods and the demand of the new platform,this thesis discusses the design and implementation of the new solution of a user behavior data management platform,which is called YueWen intelligent user data mining platform service(YUDS).The system is composed of three main modules-search module,storage module and update module.Through these three modules,the system provides a unified data interface to each business party,and provides engineering foundation for mining perfect user profiles.The search module receives user requests and calls the corresponding PLUGIN to implement specific data reading and processing logic.It manages the Plugin that has been registered on the platform and is the portal of the system.The storage module is mainly responsible for the creation of logical tables and the management of mapping between logical tables and physical tables.And it masks the underlying database differences and provide two copies of offline/real-time data to users at one time.The update module runs on the streaming computing platform Flink,subscribes to the target behavior log from Kafka and selects the appropriate Plugin to update the underlying data.The main work of this thesis includes the following four points.Firstly,it analyzes the company's original data management,introduces its new user data platform from original system's shortcomings,and analyzes the advantages of the new platform.Secondly,to solve the problem that China Literature has a wide variety of kinds of user behavior data and the data flow is huge,this thesis proposes a serialization data storage method by Protobuf to achieve behavior type decoupling.Beyond that,this thesis also proposes a way to produce plugins by providing a basic toolkit to solve the problem that different business lines have different reliance on user behavior data.Thirdly,for the real-time requirements of this system,this thesis uses the LAMBDA architecture,which can calculate the whole historical data in batch and merge historical data with real-time data online.Then through the demand analysis of the whole platform framework,this thesis designs and explains the whole system architecture diagram,introduces the overall system designs and the functions of three modules in detail.Fourth,based on the designs above,the specific implements of the three modules are written in detail.What's more,this chapter describes the implementation details of key techniques such as C++ simulating reflections and double buffer in Brpc.After completion,the platform integrates a large amount of data within the whole company.This platform provides a unified data solving interface for all business parties and improves the efficiency of user data management.And the company will build a perfect user profile system based on the platform to provide more accurate services for recommendation and other system.