Research On Extractive Multi-document Summarization Using Supervised Deep Learning

With the rapid development of Internet,the increasing availability of online information has necessitated intensive research in the area of automatic text sum-marization.As a technology to alleviate information overload,automatic text summarization has a wide range of applications in practice,such as automatical-ly generating search result snapshot for search engines,automatically generating s for news articles and technical articles and so on.Automatic text also acts as one of the key technologies in other applications,e.g,robotic reporters.Earliest studies on text summarization employ unsupervised techniques.Due to the lack of necessary supervision of human ingenuity,these methods are gener-ally less effective.In the 1990s,with the advent of machine learning techniques,various methods are proposed that employ supervised techniques to improve the performance of extractive summarization.The proposed methods have achieved significant better performances than unsupervised methods generally,however they all incorporate feature engineering as a necessary but labor-intensive task.Deep learning models recently show great potential and deliver state-of-the-art performance in many tasks,as well as in multi-document summarization.The attractions of deep learning are at least two-fold.First,deep learning offers the substantial advantage of eliminating the need for labor-intensive feature engineer-ing,so that novel applications could be constructed faster.Second,deep learning models are good at learning effective feature representations,which highlights the weakness of conventional machine learning algorithms to some extent.In this dissertation,we carry out a series of studies for improving the per-formance of generic multi-document summarization and query-focused multi-document summarization using deep learning models.We also propose a new framework to improve the sentence regression framework,which is one of the branches of extractive summarization frameworks that achieves state-of-the-art performances on many datasets and is commonly used in practical systems.The main research contents and innovations of this thesis are listed as follows.(1)We propose a neural network model for generic multi-document summariza-tion,which leverages contextual sentence relations to improve the performance.Most existing studies for multi-document summarization spend their efforts on model the sentence meanings.While understanding the meaning of a sen-tence is important to generate a good summary,the meaning of a sentence is not independent of the meaning of other sentences and sometimes it is incomplete without considering its relations with other sentences.This statement is hardly controversial since each sentence usually only expresses one view or states one fact,which may be hard to grasp without knowing the background reflected in the related sentences.Therefore,we argue that sentence saliency depends both on its own meaning and on relations with other sentences,especially contextual sentence relations.Contextual sentence relations refer to the relations between a main body sentence and its local context.To this end,we propose a neural net-work model,Contextual Sentence Relation-based Summarization(CSRSum),to take advantage of contextual relations among sentences so as to improve the per-formance of generic multi-document summarization.Specifically,we first use sen-tence relations with a word-level attentive pooling convolutional neural network to construct sentence representations.Then,we use contextual relations with a sentence-level attentive pooling recurrent neural network to construct context representations.Finally,CSRSum automatically learns useful contextual features by jointly learning representations of sentences and similarity scores between a sentence and sentences in its context.Using a two-level attention mechanism,C-SRSum is able to pay attention to important content,i.e.,words and sentences,in the surrounding context of a given sentence.We carry out extensive experiments on Document Understanding Conference(DUC)2001,2002,and 2004 generic multi-document datasets.The results demonstrate that CSRSum outperformsstate-of-the-art methods in terms of multiple ROUGE metrics.(2)We propose a neural network model for query-focused multi-document summarization,which employs attention mechanism to enhance the modeling of relevances between main body sentences and queries.For query-focused multi-document summarization,whether a sentence should be included in the final summary depends not only on its importance but also on its relevance to the given queries.The main body sentences that are close-ly related to the document topic might not necessarily answer the queries.On the contrary,a sentence that does not reflect the core idea of the article might properly answer the queries.There are usually multiple query sentences for each instance in the query-focused summarization task.Each query target at different aspect of the documents.Existing studies directly evaluate the overall relevance of a main body sentence to the given queries.This is not always reasonable since most main body sentences are short and usually only express one view or state one fact.On other words,most main body sentences can only answer one query.To this end,we propose a neural network model,Query Sentence Relation-based Summarization(QSRSum),to enhance the modeling of relevances between main body sentences and queries for query-focused multi-document summarization.QSRSum first uses a convolutional neural network to construct main body sen-tence and query representations.Then it uses a QSR-based attention mechanism to assign more weights to more relevant queries with respect to the main body sentences,which simulates the attentive reading of a human reader with some queries in mind.We carry out extensive experiments on DUC 2005,2006,and 2007 query-focused multi-document datasets.The results show that QSRSum outperforms state-of-the-art methods.Besides,with QSR-based attention QSR-Sum can point out which query sentence is answered by the main body sentence.(3)We propose a novel redundancy-aware sentence regression framework for multi-document summarization.Existing sentence regression methods for extractive summarization usually model sentence importance and redundancy in two separate processes.They first evaluate the importance f(St)of each sentence St and then select sentences to generate a summary based on both the importance scores and redundancy among sentences.We propose to model importance and redundancy simultaneously by directly evaluating the relative importance f(St|?)of a sentence St given a set of selected sentences ?.Specifically,we present a new framework to conduct re-gression with respect to the relative gain of St given ? calculated by the ROUGE metric.Our method improves the existing regression framework from three as-pects.First,our method is redundancy-aware by considering importance and redundancy simultaneously instead of two separate processes.Second,we treat the scores computed using the official evaluation tool as the groundtruth and find that our method has a higher upper bound.Third,there is no manually tuned parameters,which is more convenient in practice.Extensive experiments show that the proposed method outperforms state-of-the-art extractive summarization approaches.