Research On User Generated Text Data Mining Method Oriented For Product Requirements

Product requirement runs through the entire product development process of an enterprise and is the key factor for the success or failure of product development,which determines whether a product can obtain a relative advantage in market competition.With the rapid development of the Internet,user-generated text data from forums,Weibo,shopping websites,content communities,and industry vertical websites has become an important source of product user requirements.Compared with traditional requirement acquisition methods such as questionnaires and user interviews,user-generated text data has many advantages,such as less time-consuming,low acquisition cost,and large sample size.In addition,since these data come from users’ active sharing,rather than passive Q&A,they can reflect the real user requirements.Therefore,whether the value and knowledge contained in the massive user-generated text data can be mined in a timely and effective manner is the core issue for enterprises to keenly grasp the market requirements and the personalized user requirements.User-generated text data mining for product requirements mainly faces the following four challenges:(1)the richness of product types leads to the diversification of products mentioned in user-generated text data,making it difficult for product entities to be accurately identified;(2)the randomness of text content leads to fragmentation of information expressed by user-generated text data,making it difficult to effectively extract product information;(3)the diversity of expressions leads many users to express their opinions implicitly through factual descriptions that can be explained with sentiment,making it difficult to effectively mine implicit opinions;(4)the complexity of the psychological state causes the user’s emotions to change all the time,making it difficult to accurately detect the user’s emotions.The above challenges require a set of usergenerated text data mining method systems oriented to product requirements.Therefore,it is necessary to first identify the product requirements elements mentioned in the usergenerated text data,and then understand the semantic content expressed by the usergenerated text from the perspective of product requirements.Focusing on the above challenges,this thesis adopts the idea of “product requirement element recognition-product requirement semantic content understanding”,and uses machine learning,computational linguistics,network structure analysis,natural language processing and other methods to carry out research on product requirements oriented user generated text data mining methods from two aspects: product requirement element recognition and product requirement semantic content understanding.Firstly,this thesis identifies the basic semantic units of user generated text as the elements of product requirements from two aspects: product entities and fragmented product information,to provide the underlying information support for understanding the semantic content of product requirements.On this basis,the semantic content of user generated text data is extracted from user opinions and user emotions respectively.The specific research work and achievements of this thesis are summarized as follows:(1)A product entity recognition method that integrates product domain knowledge is proposed.First,a series of word-based and pronunciation-based rules are designed to expand the index range of product entities in the lexicon.Then,candidate product entities for each character of a given sentence are generated,and based on product domain knowledge,a referent graph model is used to model the global dependencies of candidate product entities.On this basis,the global dependencies of product entities is embedded into the deep learning model for Chinese product entity recognition.(2)A fragmented product information extraction method based on open semantic relationship is proposed.Combined with the open semantic relation extraction method,the product information extraction task is formulated as a set sequence generation problem.First,an autoregressive approach is used to learn the relation of information tuples.Then,an independent output gate is designed to evaluate the effectiveness of the generated information tuples.Finally,a differentiated learning strategy is designed to enable each decoder to learn the generation method of information tuples under different information conditions.(3)An implicit opinion mining method for fine-grained product features is proposed.Based on the semantic relation of product features,a hierarchical product feature category is constructed.Then,the characteristics of implicit opinions are analyzed,and a product feature-based implicit opinion pattern(FBIOP)is defined to represent users’ implicit opinions.On this basis,the FBIOP is applied to analyze the sentiment polarity of implicit opinions at feature level and comment level,respectively.(4)A multi-label emotion detection method for continuous clauses is proposed.The characteristics of user’s expressed emotions are analyzed,and the concept of user’s emotion transfer is defined.Then,on this basis,this thesis uses deep learning technology to predict the discrete distribution of emotions in continuous clauses by modeling implicit and explicit transfer relations of emotions on semantics and labels respectively.In addition,a method to vectorize emotions is designed when predicting emotions,and 3D visualization of different emotions is achieved through dimensionality reduction.The research results improve the method system of the basic work of text mining such as named entity recognition and open information extraction,while enhancing the comprehensiveness of user opinion mining and enriching the theoretical system of text emotion detection,which has certain management and practical value for the acquisition of product user requirements.In the future research,more in-depth exploration and research on the acquisition and integration of product requirements need to be carried out in cross modal data association analysis,cross language requirements model modeling,and design of real-time dynamic knowledge service scheme.