Design And Implementation Of The Weibo Statistial System Based On Hadoop

With the development of the Internet,network data is showing exponential growth,big data contains great commercial value,causing widespread concern from the enterprises,meanwhile,with weibo's products line constantly enriching,every product department will need to focus on more extensive and detailed user data.The company puts forward promotion strategy about headline,camera and other small APPs,in order to better cooperate with the company to develop a number of small end apps requierment,we put forword to build statistical platform of Weibo that provides statistical data and analysis function for other products division,helping products division understand and analyze their user data,and providing data persentation and data analysis capabilities to the third-party customers.Firstly,this essay introduces the developmental status of big data based on Hadoop technology at home and abroad,then it analyzes the need and system design to Weibo statistical system.The system display is used in the php,combines html?css ? Java Script and other languages to enrich and beautify the page effect.Data calculation is based on background of the hadoop technology.Synchronized mobile Terminal is handled and calculated by M/R,the result is stored to corresponding hive table.With shell,Perl,Python and other scripting to handle and calculate the data of hive table.Then we design a task scheduling system to achieve task scheduling,task dependent.task operation monitoring and other functions,and reach the porpose which are automatic scheduling tasks,easy configuration and real-time monitoring.Finally,Weibo's statistical system is tested,including functional test and the performance test.the test result shows that the Weibo's statistical system meets the function requirment,performance requirment,non-functional requirements in requirement anylysis.