Design And Implementation Of Core Functions Of Weibo User Attribute Information Mining Platform

As a social network-based information sharing, dissemination and data acquisition platform, Weibo is an important channel for people to get information, disseminate information and perceive the society. Meanwhile the data of Weibo has features of big data which are volume, velocity, value and variety. As a result, the data of Sina Weibo can be used as the foundation of Weibo user attribute information mining platform. A comprehensive and base platform for big data study will be built which could get Sina Weibo data, process big data and mining user attribute information. The research of this paper includes following three aspects.First, this study will design a special web crawler system for Sina Weibo, which is a good maintainability, strong robustness, highly intelligent reptile and will solve the fragmentation problem of the old Weibo reptile. Besides, the crawler system can provide simple and efficient data acquisition interface.Second, during data processing, fully taking into account of the big data and scalability, the Weibo platform will combine the existing data in a relational database with Hadoop distributed storage to develop and set aside a unified and complete data interface to provide basic data processing and analysis service.Finally a visualization Weibo platform for user attribute information mining is designed and the platform is B/S structure. This platform will use browser to present data in the front-end and use Weibo reptile system, relation database and Map/Reduce to acquire, process and analyze data in the backend. And the proxy interface will be used to implement transparent interaction between web server and background data, as well as between back-end and front-end.The data-driven Weibo platform achieved Weibo data acquisition, analysis and results visualization. According to the difference in the source and handing of data, the Weibo platform is divided into three modules to build the framework of the whole system. And the same time, demands in this paper are not for a particular application or a particular user, but for an ordinary data acquisition, analysis and display. And the demands intended to provide ensure for the development of a unified, systematic interfaces and services. Eventually, the platform in this paper will be a broad-based platform for Weibo big data research and new demand could be achieved by the existing ordinary interface and service.