Design And Implementation Of Data Analyzing System Of Advertising Delivery

In the current Internet context,as the coverage rate of private computers and mobile phones increases and the technological become better and better,the price of the network is decreasing very fast.People almost search on the online every day and carry out various activities.And my company,Meituan receives a lot of visits from people,so there will be a large amount of access data,so how to use and store these huge amounts of data efficiently,and then apply to various business-friendly businesses has become an urgent problem to be solved.The advertising data analysis system,as the subsystem of the DMP system(Data-Management Platform),is a very popular system among the large and medium-sized Internet companies.This system is a very important part of the RTB(Real-Time-Bidding)-DSP(Demand-Side-Platform)system.This system needs to accomplish a lot of tasks.It not only processes a large amount of data whici is from the Meituan APP,but also processes the data from the Meituan DSP system.In addition,after the data processing is completed,the system needs to be able to display and export the data.And then the data can support the work of data analyzing group,BI group and algorithm group.In the whole development period of the advertising data analysis system,I mainly participated in the requirement analysis,design,development,testing and maintenance work.My work is as follows:(1)Participated in and completed the requirement analysis of the entire system,put forward all the links and problems that I thought of,and try to meet all the needs from various groups.(2)Participated in and completed the outline design of the whole system,and finally determined that the functional modules of the system are divided into data acquisition module,data processing analysis module,data display module and data export module.(3)Cooperated with two other developer to complete the detailed design and implementation of each module of the system,including the detailed design and implementation of the data warehouse data acquisition sub-module,the detailed design and implementation of the real-time flow log analysis sub-module,and the details of the DSP log acquisition sub-module Design and implementation,detailed design and implementation of database data acquisition sub-module,detailed design and implementation of DSP log analysis sub-module,detailed design and implementation of data warehouse data cleaning sub-module,detailed design and implementation of cube production sub-module and Detailed design and implementation of the data display module and data export module which is relatively simple.(4)Participated in and completed the testing of the whole system.(5)Responsible for daily maintenance and update of the entire system,including the preparation of new ETL,the addition of new topic data,the implementation of daily automated scheduling,and so on.In the process of implementing the ad serving data analysis system,most of the technologies used are related frameworks of the Hadoop ecosystem,namely Mapreduce,Hive,Spark,Kylin,etc.The development languages used are Java,Shell and Hive Sql.The project is currently online,and has been updated and iterated many times.It is a complete system and it plays a very important role in the group.