Implementation And Application Of Opinion Extraction Based On Massive Data

With the popularity of the Internet, people’s consumption habits are gradually undergoing changes. Savvy consumers often first gather relevant information on the Internet before buying, including information of products or services and other consumers’ comments, in order to further guide the consumption. However, to extract and summarize other consumers’ opinion from a vast ocean of data is obviously a time-consuming matter. Opinion extraction technology designed to achieve machines automatically extracting the opinion target and sentiment of the reviews, one can help consumers make better decisions faster, and it also helps manufacturers and other companies better monitor their own products.In this paper, the main research works include the following:(1) Study a variety of methods including word-vector model, text clustering and rules for effective preprocessing.(2) By building a sentiment lexicon and product features lexicon, this paper use dependency parser to parse the reviews and construct grammatical structures path of product features words and sentiment words. Besides, this paper trained ten million microblog data that labeled automatically by emoji to effectively identify opinion from reviews.(3) By extracting several features in the text, use the SVM classifier to filter the correlation between target products and the extracted opinion, which further improve the accuracy of the extraction.(4) Based on the above extraction method, this paper implemented a massive data-oriented multi-domain word-of-mouth monitoring system, which includes a web crawler module based on a distributed computing framework called Spark, a extraction module based on Hadoop RPC and a web-based graphics products show module which to help users make in-depth analysis intuitively.The innovative work of this paper as follows:(1) A variety of methods for text preprocessing, including the use of the word-vector model to achieve name disambiguation algorithm of some products;(2) The dependency parsing is applied to the opinion extraction by mining multiple high accuracy extraction templates. And the use of SVM classifier to classify the correlation between the opinion and the target product;(3) Implementation of a massive data-oriented multi-domain word-of-mouth monitoring system.The word-of-mouth monitoring system based on opinion mining technology of this paper has been used in many top brand companies for their market research. And the effect has been unanimously praised.