Design And Implementation Of Data Management Platform Based On Spark

With the rapid development of the Android operating system,Android smart phone mobile APPs have been showing explosive growth, the user’s leisure time is also transferred from the PC to the mobile terminal. APP running users produce a large amount of log data and the data is becoming an important source of user interests mining. Mine the user preference and interest,take targeted advertising and content distribution, and improve marketing effectiveness.This thesis introduces the relevant background of the topic, then insights into the field of data processing related technologies, including distributed computing framework Spark, distributed file system HDFS, etc., then introduces the algorithm of user portrait system and recommendation system. Finally, based on Spark, we design the big data management platform DMP, which is a highly available data management platform using data processing techniques whin Spark ecosystem, and we also base on this platform to recommend APP for users.The main results of this paper include:(1) The design of DMP based Spark. With the actual demand of internal operations, and the big data processing technologies, we design and implement the big data management platform based on Spark cluster.(2) The design of data warehouse. We use Spark programming model to do warehouse ETL automately.(3) The design of user-portrait system. We mark each user with detailed dimensions of attribute labels and interest labels, then store the tagsinto NoSQL database.Finally, build a scalable user portrait system.(4) The design of recommendation system. We use the historical duser data to train the recommendation algorithm and build recommendation system.