A Study And Implementation Of Automatic Evaluation Methods For English-Chinese Machine Translation Systems

Machine Translation Evaluation(MTE) plays an important role to the development of Machine Translation technology and its market prospect. MTE methods can be divided into two categories: human assessments and automatic evaluation, where human assessments is to evaluate the candidates produced by MT systems with the help of humans'referencing to some criteria, while automatic evaluation is to evaluate it by computer, requiring that its evaluation results accord with human assessments as far as possible. The main work of this paper is to analyze automatic English- Chinese MT evaluation methods in details.There are many traditional methods for MT automatic evaluation, three main of which include: BLEU,NIST,WER. The basic idea of BLEU is to compute evaluation scores by counting the number of n-gram co-ocurrence between references and candidates given by MT systems. NIST is another statistical method counting n-gram co-ocurrence based on BLEU, which assigns a higher weight to a more informative n-gram co-ocurrence with a less number occurring in references. The essence of WER is to automatically evaluate the performance of MT systems by a technique of normalizing the edit distances between candidates and their corresponding references. Although the three methods, BLEU and NIST of which have been accepted as international standards, can give automatic evaluations with a satisfactory correlation to human assessments, the scores of them have some large different values from that of human assessments. Thus the NED/NES method is proposed to solve this problem. NED/NES is also based on the concept of edit distance with no less correlation to human assessments than some other methods, but the normalizing technique used by it is more reasonable than that by WER in theory, and the scores computed by it is closer to that assessed by human in application.Based on the above four methods, an English-Chinese MTE system ---- ECEvaluation has been designed and implemented. In the system, there are two modes including"outline"and"stochastic"which can be chosen to produce test sets. Also, the candidates produced by MT system for the test sets can be evaluated from the"character"or"word"point of view, providing more information about evaluation...