Shanghai Composite Index Prediction Based On Today's Headlines

With the advent of the era of big data and the development of machine learning technology,the analysis of unstructured data to predict the trend of stocks has been actively studied.The stock market is affected by many factors,and predicting the direction of the stock market has been one of the most widely investigated and challenged questions by investors and researchers.As a statistical indicator that measures the overall trend of stocks listed on the Shanghai Stock Exchange,the Shanghai Stock Index plays a very important role in the stock market.It is very important for investors and shareholders to predict changes in the Shanghai Stock Index.It is found that there is an obvious correlation between Internet stock market information and stock price fluctuation.Many previous studies have also shown that stock market news reports are highly correlated with stock price movements and have valuable information,such as news reports and company reports.By mining the information contained in the Internet stock market news and analyzing the correlation with the stock price trend,we can predict the stock price trend through this correlation.Therefore,it is very important to extract valuable information and find out the relationship between the extracted information and the stock market.In domestic and foreign literature,on the other hand,there are a large number of literature research is join the text sentiment indicators,in the original technical indicators on model accuracy of ascension,and in this respect more literature is focused on discussing how to better extraction of news text represented by the sentiment,by giving every news,message or tweet mood scores are calculated respectively;On the other hand,various machine learning models are used to predict the stock market on technical indicators.However,few people have studied the direction of sse's change in an end-to-end way through big data text mining.In the era of information explosion,information redundancy and information deviation have a great influence on people's sensory decision-making.In the era of rapid growth of big data,what news content or news recommendation can quickly catch people's attention is also a hot topic.Headlines today as today's hot to recommend search engines,data mining,it is based on analysis of the user's interests and hobbies,etc.,undertake personalized content recommendation,praised by the masses of users,each big section in active users is quite large,so in today's headlines such big platform,a huge collection of news every day how much impact on the Shanghai composite index,a proposition that is worth studying.So for this research thesis,in this article,is proposed to study how to make use of today's headlines-financial stocks rich text information to predict the stock market,this paper prepared using text mining and machine learning methods to analyze today's headline news text stock plates,first through the inside of the python jieba Chinese word segmentation packages,on today's headline news text segmentation,feature extraction,through the pageRank algorithm to get the text key words,and extract the keywords frequency,the algorithm for 678 keywords as the corpus in this paper.Then,the top 40 keywords were selected through the initial feature filtering and dimensionality reduction by chi-square value.Finally,the feature combinations with the best effect of the model were screened by RFECV,and the selected words were used as the input of the final model.In this paper,using logistic regression,support vector machine(SVM),CART decision tree,random forests,XGboost and GBDT,through to the final assessment results of the model analysis of variables on the Shanghai composite index change direction prediction ability,and the stand or fall of model,in order to better understand today's headlines stock news content for China's stock market prediction or explanation ability,at the same time according to different model predicted results to build the stock market trading strategy,will trade income compared with same period index returns,analysis under different model predicted results can get excess returns,finally under the established trading strategies,GBDT performed best,with a yield of 10.59%,outperforming the Shanghai composite index by 5.05%over the same period.