Design And Implementation Of Data Platform

In recent years,the development of technology has gradually entered the era of big data.The big data platform construction has become the needs of various technology companies,internship companies also want to establish a big data platform,in order to solve the massive data storage bottleneck.The project comes from my internship company during the graduated period,which target is to consolidate all the business data in the company,establish the unified data center,support on-line or off-line data for all of the business line,and turn into a unified platform for data exchanging and providing in the company.The project features include six modules:data acquisition;reporting system;service monitoring;log analysis;data search and analysis;user portraits.Among them,the data collected from the network crawler,business line data,log files.The reporting system is responsible for the statistical display of business indicators.Service monitoring includes cluster and data monitoring.Log analysis is the official statistics of the official website.Search analysis utilizes distributed multiuser capabilities to search engine Elesticsearch for processing large-scale data,currently in the development phase.User Portrait is based on the user's behavior data for users to label.During my internship,I was involved in the discussion of some of the features of the Big Data Platform,the development of functional modules,and the testing work.The specific content is as follows:(1)Participated in the needs of BigData platform analysis,combined with the demand division function module,involved in the development of data platform requirements document.(2)Participated in the data acquisition function module development,mainly responsible for the log file collection service framework to build and develop,business database synchronization work.(3)Participated in the reporting system module.Reporting system needs to configurate with a web front-end which include configurating with multi-table connection,table merging and so on.Based on the configurated file,a new report created by the computation in the background.The data of the new report will display by Data-cube system.(4)Participated in the development of the server monitoring module.The function of this module includes monitoring of cluster,changing of table structure and the judgment of table data.If an exception occurred,the system will inform the related person who is in charge of this function by email.(5)Participated in the development of log statistics and analysis module.The main function of this module is to count the visitor amount included PV(Page View),UV(Unique Visitor)and so on which sent by the company's official website.(6)In charge of designing in some part of database and the function module testing in the last period of development.This project has running online,which provide a stable data service for the company and a lot of support for all kinds of business.It improved the working efficiency greatly and made a strong insurance for the company business developing.