| With the rapid development of Internet technology, the technology of the monitoring of the network public opinion has flourished, the amount of data over the network public opinion monitoring system gained rapid expansion, especially the Weibo public opinion data, with the feature of the faster growth of rate data, viratity of data types, non-structural data coexisting with structured data and so on. All these features make how to store and manage the big data of Weibo become a challenge in this field.This paper combines the public opinion data of Weibo with its application characteristics, applying mainstream of large data storage and processing -Hadoop, designing and implementing a storage platform of Hadoop for the storage of the public opinion of weibo. The concrete work and contributions of this paper are as follows:(1) This paper analyze the storage needs of the Hadoop storage platform of the Weibo public opinion and the characteristics of Weibo data, presented a Hadoop storage platform based on a micro-Bo public opinion. It designs the network-based structures and systems of the Hadoop storage hardware subsystems based on Weibo detailly,constructing experimental hardware environment which consists of 13 nodes.(2) This paper design and implement the Hadoop storage software subsystems based on Weibo, including the configuration and deployment of Hadoop system, Mysql data table which manages the Weibo metadata; Hbase data table which manages the unstructured data of Weibo.(3) This paper develops a software model of search and retrieval interface of Weibo data, tests and analyzes the performance of the Hadoop storage platform based on Weibo public opinion.Experimental results show that the platform designed on this paper satisfies the analysis application on the network public opinion. The system constructed by this paper has been used in Tsinghua University "distribution and share platform of big data on internet network" and other projects. |
PDF Full Text Request