Prediction Of Stock Market Using Web Data Mining Based On American Market

The hypothesis that investors should be rational supported by the theory of EMH (Efficient Market Hypothesis) has been suspected by the opinions supported by the theory of Modern Human Financial. And in empirical studies, it is confirmed that investors’emotion or subjectivity has a remarkable effect on stock prices. Early in relative studies, the challenge focused on finding proper market or social variables to reflect the polarity and level of such sentiment to predict effectively the direction and level of stock movement. With the coming of Informational Age, sources of data are becoming more diverse, and technology of data mining is also becoming sophisticated. All these stimulate relative studies further focusing on specific intensity and volatility of market returns instead of stock prices direction and level. Thus the critical difficulty faced with scholars lies in how to quantify such emotional impulses derived from investors’social psychological characteristics. This article also focuses on how to implement a complete and effective quantitative analysis on public mood in market to improve the accuracy of prediction on stock movement.Though effective methods of quantifying market sentiment make it possible that the concept of Socioeconomics is introduce in the traditional financial market analysis, how to realize this transformation from analog information composed of investors’ subjectivity to digital information which can be utilized by mathematic or computer simulation model is a new-bora challenge. In addition, how to map emotional variables with stock market features is also a cutting-edge problem scholars confront. Especially, as Internet median platforms and social networking sites develop dramatically, human beings must face vast networking data, and it is obviously beyond person’s capacity of collecting and processing data, even traditional optimization algorithms in computer science can’t satisfy increasing demand of application in real issues. Responding these needs in practice, Internet Crawler technology and Machine Learning Algorithm improve greatly and develop various methods revolving around specific demand of practical problems.This article therefore proposes a systematic quantitative trade strategy utilizing market sentiment information of Internet, including news articles from financial media and statistical data from Internet service providers. Aiming at this investment decision-making way, the whole work consists of three components:1.Crawling web information from specific media’s website and filtering rough data to effective examples like html and txt so that can be used to extract public mood information. Additionally, this part also includes classifying textual examples into clusters according to time distribution.2. Quantifying media textual information and transformation of public mood and concentration. This work is composed of two part. One includes textual syntactic analysis on and extraction of characteristics from news articles. The other is processing of statistical data which capture the movement and volatility of investors’concern, like SVI (Search Volume Index), generated by Internet Service Providers.3. Joining psychological emotional factors and financial market factors together. On the basis of traditional financial market model based on the concept of risk premium, like Fama-French Three Factor Model, a new financial model is proposed, which covers non-market indices so that makes up the defect that only use pure market factors, like general economic indices or firm-specific ones, to describe the status of market. Meanwhile, the article adopts linear and non-linear methods to map public mood with stocks movement and compares their respective effects. In linear one, a linear multiple regress model, which is also a modified Fama-French Three Factors Model, has been proposed. In non-linear one, machine leaning algorithm has been introduced to directly map textural linguistic characteristics with market movement in order to describe and predict stock market more efficiently and accurately.These three parts cooperate with each other to form a systematic investment decision-making strategy. This study implements the application of two kinds of mutual Internet information which reflect public mood and expectation on market from two different aspects. Distinctly, the core concept of this study has a dramatic significance in practical application of human factors in financial market analysis.