A recent study conducted by researchers at The University of Granada’s School of Translation and Interpretation attempts to analyze and evaluate the results of machine translations done with popular online tools such as Google Translator, Promt, and WorldLingo. The study was published in this month’s issue of Translation Journal, and it raised interesting questions for me about the possible uses for online machine translation.
Looking at the findings, it should come as no surprise that all of the machine translation tools produced poor results in terms of the number of errors, or that after the translations passed through a round of human editing, the number of errors were drastically reduced. What is interesting, though, is that certain online tools performed better than others, and specific language combinations produced varying results. The graph below shows results from German into Spanish (the researchers used EvalTrans Software). The best translation machine is the one showing the lowest word error percentage (WER). Check out the study for more charts and an explanation of the sentence error rate (SER).
Doctors Lola García-Santiago and María-Dolores Olvera-Lobo do a thorough job of laying out the methodology that they followed, and of acknowledging the difficulties inherent to such studies. They write that,
Evaluation of machine translation is an unresolved research problem that has been addressed by numerous studies in recent years. The most extensively used assessment tools are classified into two major groups: automatic objective methods, and subjective methods (Tomás, Mas & Casacuberta, 2003). The objective evaluation methods compare a set of correct translations of reference against the set of translations produced by the translation software under evaluation. The units of measurement most often used work at the lexical level, comparing strings of text.
