Design And Implementation Of Weibo Analysis System

With the development of Internet, social networks and human life is becoming more and more inseparable. Weibo, as a typical representative of social network, is attracting more and more people. This leads to the fact that there are so much data on weibo, that people can hardly find the information they are interest in.The main purpose of weibo analysis is to filtering and induction the data, in order to help users finding the information they need as soon as possible. In this paper, we just implement such a weibo analysis system. This system focus on users in the same organization, analyzing the characteristic of the users.At first, we design the overall framework of the system. It is divided into several major modules, such as data acquisition, data storage, data analysis. We improve system stability and extensibility as far as possible. We improved the ability of handling big data by designing the system on Hadoop.In the data acquisition module, we combine web crawler and weibo API, and implements a functioning weibo crawler, completed the data storage, update, and other functions. In data storage module, we has carried on the design to the database table, in order to achieve the balance between the system efficiency and storage space.Data analysis module, we use three levels, all users in the organization, users in the same civil society and one user’personal contacts. We do the analysis on the user association, text, LBS(Location Based Service), and other fields. The main modules include community detection, influence calculation, hot topic detection, keyword extraction, weibo position inference, etc.Weibo position inference module is the most important feature in this paper. This part is mainly to solve the problem that there are too little location information on weibo. In this respect, we use the space and time distribution of the words to identify local words. We all used the behavior and relation of users to improve location inference. At the same time, through focus on user behavior and user relationship, greatly improve the position of the effect. At the same time, in order to better solve the problem of weibo location type annotation, we also studied the short text classification algorithm, and achieved good effect.