Design And Implementation Of NetEase Mobile Big Data Support Platform Based On Spark And Hive

For Internet products,accurate decision-making and operation determines the prosperity and decline of products.At present,the strategy and operation of Internet products mainly rely on the experience of product team and operations team,but those people who really can do precise decision is very rare.The traditional operation method is not only cost a lot of money,but also exsits problems like low user stickiness.NetEase as a pioneer of the Internet,all of its business lines have tons of data.However,the data analysis of each business line is scattered,the scope of application is narrow and lack of generality.Therefore,the company urgently needs a platform to integrate the internal data resources for the relavent department to provide accurate data analysis support service.Under this background,the mobile big data support platform came into being.Its appearance provides a good solution to overcome the limitations of traditional decision-making and operational means,and provides a good solution to improve group resource reuse rate.The platform is based on distributed component development.The component it used including Kafka,HDFS,Spark,Hive,HDFS Connector,etc.At the same time,the platform combines Spring,MyBatis and other Java Web framework technologies to support the platform background servies.The platform is composed of infrastructure subsystem,data analysis subsystem and data mining subsystem.During the internship,the author is mainly responsible for the design and development of the data analysis subsystem,which includes 5 modules:user behavior analysis module,transaction behavior analysis module,page thermodynamic diagram module,path analysis module and assistant services module.Among them,the functions of the first four modules are completed by the data analysis part and the background service part,and the functions of assistant module is completed independly by the background service part.Data analysis uses big data analysis technology,and the background service part uses Java Web technology.This paper firstly introduces the key technologies used in the development process of the platform.Secondly,it describes the overall function of the platform and focuses on the functional modules that the author is responsible for.It also analyzes the non functional requirements of the platform.Then it introduces the design of the system architecture,function architecture and the database design.After that,it describes the detailed design and implementation process of the platform,and shows the implementation diagram of each module.Finally,the test and verification methods of this platform are described.Mobile big data support platform provides a simple and convenient access point and accurate data support services for many NetEase's products,it improves the reusability of the company's internal data resources,and improves the work efficiency of product department,operation department and other related departments.At the same time,it provides data sources for data mining teams,helping them to further explore the value of data,which helps improve the imcome of the department and whole company.