BLEU

BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Quality is considered to be the correspondence between a machine’s output and that of a human: “the closer a machine translation is to a professional human translation, the better it is”.^[1] BLEU was one of the first metrics to achieve a high correlation with human judgements of quality,^[2]^[3] and remains one of the most popular.

Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good quality reference translations. Those scores are then averaged over the whole corpus to reach an estimate of the translation’s overall quality. Intelligibility or grammatical correctness are not taken into account.

BLEU is designed to approximate human judgement at a corpus level, and performs badly if used to evaluate the quality of individual sentences.

BLEU’s output is always a number between 0 and 1. This value indicates how similar the candidate and reference texts are, with values closer to 1 representing more similar texts.

Notes

^ Papineni, K., et al. (2002)
^ Papineni, K., et al. (2002)
^ Coughlin, D. (2003)

References

Papineni, K., Roukos, S., Ward, T., and Zhu, W. J. (2002). “BLEU: a method for automatic evaluation of machine translation” in ACL-2002: 40th Annual meeting of the Association for Computational Linguistics pp. 311–318
Callison-Burch, C., Osborne, M. and Koehn, P. (2006) “Re-evaluating the Role of BLEU in Machine Translation Research” in 11th Conference of the European Chapter of the Association for Computational Linguistics: EACL 2006 pp. 249–256
Doddington, G. (2002) “Automatic evaluation of machine translation quality using n-gram cooccurrence statistics” in Proceedings of the Human Language Technology Conference (HLT), San Diego, CA pp. 128–132
Coughlin, D. (2003) “Correlating Automated and Human Assessments of Machine Translation Quality” in MT Summit IX, New Orleans, USA pp. 23–27
Denoual, E. and Lepage, Y. (2005) “BLEU in characters: towards automatic MT evaluation in languages without word delimiters” in Companion Volume to the Proceedings of the Second International Joint Conference on Natural Language Processing pp. 81–86
Lee, A. and Przybocki, M. (2005) NIST 2005 machine translation evaluation official results
Lin, C. and Och, F. (2004) “Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics” in Proceedings of the 42nd Annual Meeting of the Association of Computational Linguistics.