Research On Automatic Machine Translation Evaluation With Documental Information

In recent years, thanks to the research and development of automatic evaluation, ma-chine translation has made significant progress. But it is constrained by the limitations ofmetrics for evaluation either. In order to combat negative influences of existing evaluation,this paper proposes a new automatic evaluation method and especially describes how tointegrate documental information into current automatic evaluation metrics.Firstly, a new method—Phrase-Based Evaluation (PBE) is introduced, which useslinguistic phrase as the basic unit of evaluation. PBE consists of three modules: assigningphrase weight, calculating phrase similarity and finding the maximum similarity map. Inthis paper, the basic weight function includes three diferent implementations referring tongram, tf.idf and C-value respectively. The similarity function is based on WordNet and theKM algorithm is used to find a maximum similarity map between two sequences of phrases.Secondly, this paper investigates how to integrate documental information into existingautomatic evaluation metrics. Topic model is first introduced. According to LDA, a pop-ular method for obtaining topic model, each document and word have a distribution overdiferent topics. Then topical distribution of a phrase can be derived. And finally, topicaldistributions are integrated into weight function and similarity function of PBE within asimple framework.The second kind of documental information is lexical cohesion, which reflects the co-herence of a document. Based on LC method, the Weighted LC (WLC) is proposed in thisarticle, which assigns weight for a word by PageRank algorithm running on word graph ofdocuments. Further, a new method named pos-WLC which biases PageRank algorithm towords with specific POS tags is proposed. And this paper further shows how to combine theevaluation of lexical cohesion with PBE and other mainstream automatic evaluation metrics,in order to help these methods to evaluate translation quality at document level.Experiments on MTC2(LDC2003T17) and MTC4(LDC2006T04) show that PBEachieves the higher Spearman correlation with human judgements than the most popularmethod BLEU. When topic model is incorporated, performance of PBE on MTC2and some systems of MTC4is further improved. Compared with LC, WLC and pos-WLC are alsohave higher Spearman correlation at document-level evaluation. And after being combinedwith others metrics, in most cases, both contributes more efort to the integrated performanceof evaluation than LC.