Neural Network Based Discourse Coherence Modeling

Coherence means sentences kept in a specific order which makes texts meaningful both logically and syntactically. For this kind of text, a high performance discourse coherence model is important for natural language processing and generating tasks. For a text with well-written sentences, it will be totally unreadable if the sentences swap randomly without changing word order.The discourse coherence modeling is widely used in natural language processing and generating applications. But the existing discourse coherence model cannot works well. For all these models only capture the cross-sentence information but ignore the word relationship within sentence. This results the unsatisfying performance of these discourse coherence model.To do discourse coherence modeling better, we propose a maximum entropy based discourse coherence modeling approach. In contrast to the previous methods, our model uses only lexicon features to do discourse coherence modeling discarding the other features such as syntax. It examines the probability of doing discourse coherence modeling only with lexicon features.To optiminze our model, we also extend our maximum entropy based discourse coherence model to a recurrent neural network based sentence-level language model. The extended model has a better performance on doing discourse coherence modeling and it would be easy to get a real-valued feature vector from the recurrent neural network.We then propose a novel hierarchical recurrent neural network language model to capture the coherence both within one sentence and between sentences in a document. The hierarchical recurrent neural network language model integrates the sentence history information into the word-level language model to predict the word sequence. A two-step training approach is designed.At sentence-level, we examine the model by standard sentence ordering scenario. At word-level, we use perplexity to evaluate our model. We also conduct a Chinese-English document translation reranking task. All these show that our model outperforms the state-of-the-art baseline systems.