Sina Weibo Data Mining To Predict The Shanghai Composite Index

The social network service has been developed very fast in recent years, and in China, Sina Weibo is playing an important role for people getting and sharing information. Sina Weibo offered a large amount of direct or indirect raw data and thus was selected as the data source of this study. In this article, enough text data were crawl from Sina Weibo by web crawler, and then were used to forecast the trend of Shanghai securities composite index, and thus helping the investors to make better decision.After analyzing and solving problems like login simulation, advanced search, IP visiting frequency limit and text processing, in this article, a web crawler was made to collect enough data from Sina Weibo.With all data prepared well, a forecasting model was built using Sina Weibo contents to predict the trend of Shanghai composite index. The main algorithms used are K-means clustering and Back-propagation artificial neural network.Creative ideas include:1. In this article, Sina Weibo contents were used to forecast the trend of Shanghai composite index, and this is relatively new in China.2. To the Sina Weibo content crawler, distributed system mechanism was involved in to solve the visiting frequency limit.3. Using the advanced search function from Sina Weibo to get data set with time and theme characteristics.4. By modifying both the Back propagation artificial neural network and the K-means clustering algorithms according to the characteristics of the data set, the forecasting precision was improved.