Research On BERT-based Neural Ranking Models

With the rise of the era of big data,data has become a new energy driving economic development How to accurately retrieve user demand data from massive data and dig out valuable information from it has become a research hotspot in the field of information retrieval.At present,word embeddings are widely used as the input of neural network models in various natural language processing tasks.Experiments show that the use of word embeddings,compared to the bag-of-words model,can significantly improve the performance of these tasks.However,traditional word embedding approaches cannot effectively model the context,and solve the problem of word ambiguity.In 2018,Google released the pre-trained language model BERT,which can disambiguate words by encoding contextual semantics.Inspired by the excellent performance of BERT in many natural language processing tasks,we conduct a systematic study on how to use BERT to improve the performance of the information retrieval system.The main work includes the following two aspects:First,we use BERT to build a representation-focused neural ranking model.We first use BERT to encode the contextual word embeddings of queries and documents respectively,and then input them into the neural ranking model constructed by Transformer to get the relevance score of the query and document.We call it ATER(ATtentional Embeddings for Ranking).We conducted experiments on the classic information retrieval datasets Robust04 and MQ2007.The experimental results show that the neural ranking model based on the BERT word embeddings performs better than the traditional information retrieval model in terms of P@20 and NDCG@20,which prove that the model based on the context-sensitive word embeddings can construct a more accurate semantic representation of text.Secondly,we use BERT to build an interaction-focused neural ranking model.Following Yang W et al[1],we first use BERT to capture matching patterns between the query and document paragraphs,and then input it into a linear classification layer to get the relevance score of the query and document.We conducted a lot of experiments on Robust04 and MQ2007 datasets.The experimental results show that,compared with the traditional retrieval model BM25 and the representation-focused neural ranking model ATER,the neural ranking model based on fine-tuned BERT achieve better retrieval performance,which indicate that the interaction-focused neural ranking model can more accurately capture matching patterns between queries and documents.