Research On MT Automatic Evaluation Based On User Behavior

Machine translation evaluation plays a great role as "Moses" in thedevelopment of machine translation techniques. However, the performance ofstate-of-art string similarity-based MT evaluation metric is invariably consideredto be inadequate. Although, researchers have successfully applied linguisticfeatures to reinforce it, the gain of performance is limited. As MT continues todevelop, there are various online machine translation services which havebecome the routine use for web users. This process is producing user feedback ona large scale. User feedback has often been proposed as a solution for improvingmachine translation systems. The basic motivation is that users will be able tomake disambiguating choices, take post-edit actions which fall outside of themachine translation system’s capabilities, or supply extra-linguistic knowledgenecessary for language analysis or language generation. Until now, relativeresearches on utilizing expert feedback in MT automatic evaluation havedemonstrated the profit of user feedback while ignoring the exploiting the fuzzyfeedback from ordinary users. With the help of community wise and qualitycontrol, ordinary user feedback seems to be a good substitution of expertfeedback. Whether or not it would service to evaluate machine translation outputmore user-centered and closer to expert evaluation than the state of art MTevaluation metric? How to get rid of noise, and then make use of user feedback toimprove machine translation system?Aiming at solving these problems, we explore the following aspects:1. Recording user feedback through designed user experiment with web-based toolkit.2. After investigating the relationship between user behavior and humanperception of translation quality, we delve the feasibility of machine translationevaluation modeling based on user behavior by constructing different sets of userbehavior features. Besides, we try some feature selection strategies to optimizeMT evaluation model in the guidance of analysis about individual user behavior’scontribution.3. We build a post-editing model trained on parallel corpus gained from user feedback and test its performance.It is revealed that the majority of intuitively designed user behavior featureshave a significant correlation coefficient to human evaluation. It is suggestedmachine translation automatic evaluation based on user behavior can perform asgood as many state-of-art evaluation metrics if not better than. The comparativeexperiment illuminated the performance of such model is subject to the quality ofuser feedback. Phrase-based post-editing trained on user edited version andrelevant system output can improve translation quality, while it doesn’t work allthe time.