Improvement And Optimization Of The Data Acquisition System Of The BI Platform For The Mobile Reading

As the rapid development of IT technology and communication technology, such as mobile Internet, Internet of things, cloud computing,social network and so on, the amount of data grows rapidly. Big data opens an era of large-scale collection, sharing and application of data, it brings tremendous changes and far-reaching impact to the technology of data collection and application. Big data are paid more and more attention to,and data acquisition challenge is particularly prominent, such as the various data sources, large amount of data and quick change, how to ensure the reliability of the data collection, how to ensure the quality of the data,etc. The traditional data acquisition technology can't meet the current demand, the improvement and optimization of the data acquisition system conform to the requirement of the mobile internet.The BI platform for the mobile reading is a platform that China Mobile analysis the user's reading information on mobile phone, in-depth understanding of the customer's personalized online reading needs,accurate positioning of customers, so as to recommend appropriate ways to business, and ultimately meet the needs of users. The old data acquisition system of the platform has three problems: poor data rollback, data delay,ETL plug-in development and use of low efficiency. This thesis mainly aims at the problems existing in the original business operation mechanism of the platform, and puts forward the corresponding improvement and optimization solution.The improved system can support real-time data collection of different data sources and provide cluster status monitoring function. The problem of data rollback poor is optimized in the third chapter. The method of message middleware is brought forward by the reason of the poor performance of the original system data, and then the overall design,detailed design and implementation of message middleware are expounded.In chapter 4, the problem of data delay is optimized. On the basis of message middleware, the data structure design, overall design, detailed design and realization of the optimization scheme are expounded respectively. The problem of data delay in data collection and data storage is optimized. The fifth chapter is to improve the development and use efficiency of ETL plug-in. First of all, according to the causes of problems proposed improvement program, and then, the reconstruction of data processing functions, such as transcoding, data statistics and URL parsing,are expatiated. Then, the system carries on the key function test and the performance contrast analysis. Finally, this thesis briefly summarizes the realization of the system, and prospects its future development.