Research On Computing Granularities Of Automatic Machine Translation Evaluation

The evaluation methods are a key technology which has very important significance for the machine translation research. Automatic MT evaluation metrics can play an important role in the development recycle of MT systems. Currently, many string-based metrics may evaluate the MT system output fast and simplely, but the results of the metrics have serious bias. A popular trend is combining the more linguistic informantion into the metrics; however, the linguistic informantion is at the cost of losing language independency and prevents the wide application of the linguistic motivated metrics. In fact, with more linguistic features attributed, the varieties are considered as changes of the calculation unit (or granularity) in the matching.Focusing on how to improve the accuracy and speed of automatic evaluation metrics, and to enlarge the application-bound of them, we change their original calculation granularities of the metrics, and provide a series of effective automatic evaluation metrics.Firstly, we propose the letter-based automatic MT evaluation metrics. The metrics are independent of the language, and slove the part of the word variation problems. In order to improve the performance of the metrics, we provided two metrics, i_letter_BLEU and i_letter_Recall. They both can automatically adjust the parameters according to references, and are more stable in performance of the letter-based metrics.Secondly, based on features combination metrics by machine learning, the string-based metric combining multiple calculation granularities is provided. The strategy combines the features using the SVM rank and regression models. The metrics use few features through the feature selection to obtain the comparable performance with metrics competitions in last years. The underlining is that the metric does not require any deep linguistic information, and is independent of language.At the last, we uniform the linguistic features into strings, re-examine the contribution of linguistic features to automatic MT evaluation and propose a metric based on combining linguistic granularities. The strategy still combines the features using the SVM rank and regression models. Using few selected features among the features in multiple calculation granularities, we provided metrics with lower complexity, but have higher performance, compared to classical metrics.