Design And Implementation Of Intention Recognition Algorithms Based On Vertical Search

Vertical search is a kind of search in professional field.Unlike general search,vertical search requires a deep understanding of the user's intentions and returns the information in the user's specific field.Vertical search engines need to train multiple intention recognition models for different domains because of the different search intentions in difterent domains,so the training cost of intention recognition model in vertical search scenarios is high.This paper mainly solves the problem of high training cost of intention recognition model in vertical search scenario.The author's main work includes Semantic Vector Pre-training based on SimNet and Intention Recognition Model based on Transfer Learning.(1)Semantic vector pre-training based on SimNet mainly uses massive data accumulated by search engines to pre-train sentence vectors based on semantics.The author uses the QUQ(Query to url to query)roaming algorithm,which is originated in this paper,to obtain a large number of weak labeled samples,train SimNet model through a large number of sanples,and finally generate semantic vectors according to the output of SimNet model.Experiments show that semantic vectors can express sentence semantics well and have a certain generalization.(2)In the training intention recognition model stage,the author transfers the pre-trained semantic vectors to the intention recognition model,and uses active learning to reduce the tagging cost.Firstly,samples are transformed into semantic vectors and seed samples are used to initialize the intention recognition model.Then,Best vs Ssecond-best(BvSB)algorithm is used and improved to reduce sample redundancy and select samples with high information.Finally,the intention recognition model is trained according to the sample.Experiments show that the transfer learning training intention recognition model can reduce the labeling cost by more than half,and the improved BvSB algorithm can further reduce the labeling cost and improve the accuracy of the model.The author's main research work is carried out in Baidu Vertical Industry Search Department.Baidu Vertical Search has been used in more than a dozen fields.This paper trains a high accuracy intention recognition model with less cost,which affects more than 100 million search users.