? Optimizing Non-Decomposable Evaluation Metrics for Neural Machine Translation
Journal of Computer Science and Technology
Quick Search in JCST
 Advanced Search 
      Home | PrePrint | SiteMap | Contact Us | FAQ
 
Indexed by   SCIE, EI ...
Bimonthly    Since 1986
Journal of Computer Science and Technology 2017, Vol. 32 Issue (4) :796-804    DOI: 10.1007/s11390-017-1760-9
Special Issue on Deep Learning Current Issue | Archive | Adv Search << Previous Articles | Next Articles >>
Optimizing Non-Decomposable Evaluation Metrics for Neural Machine Translation
Shi-Qi Shen1,2,3, Student Member, CCF, Yang Liu1,2,3,4,*, Senior Member, CCF, Mao-Song Sun1,2,3,4, Senior Member, CCF
1 Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China;
2 State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing 100084, China;
3 Tsinghua National Laboratory for Information Science and Technology, Tsinghua University, Beijing 100084, China;
4 Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University, Xuzhou 221009, China

Abstract
Reference
Related Articles
Download: [PDF 311KB]     Export: BibTeX or EndNote (RIS)  
Abstract While optimizing model parameters with respect to evaluation metrics has recently proven to benefit endto-end neural machine translation (NMT), the evaluation metrics used in the training are restricted to be defined at the sentence level to facilitate online learning algorithms. This is undesirable because the final evaluation metrics used in the testing phase are usually non-decomposable (i.e., they are defined at the corpus level and cannot be expressed as the sum of sentence-level metrics). To minimize the discrepancy between the training and the testing, we propose to extend the minimum risk training (MRT) algorithm to take non-decomposable corpus-level evaluation metrics into consideration while still keeping the advantages of online training. This can be done by calculating corpus-level evaluation metrics on a subset of training data at each step in online training. Experiments on Chinese-English and English-French translation show that our approach improves the correlation between training and testing and significantly outperforms the MRT algorithm using decomposable evaluation metrics.
Articles by authors
Keywordsneural machine translation   training criterion   non-decomposable evaluation metric     
Received 2016-12-20;
Fund:

This work is supported by the National Natural Science Foundation of China under Grant Nos. 61522204, 61432013, and the National High Technology Research and Development 863 Program of China under Grant No. 2015AA015407, also supported by the Singapore National Research Foundation under Its International Research Centre@Singapore Funding Initiative, and administered by the IDM (Interactive Digital Media) Programme.

Corresponding Authors: Yang Liu     Email: liuyang2011@tsinghua.edu.cn
About author:
Cite this article:   
Shi-Qi Shen, Yang Liu, Mao-Song.Optimizing Non-Decomposable Evaluation Metrics for Neural Machine Translation[J]  Journal of Computer Science and Technology, 2017,V32(4): 796-804
URL:  
http://jcst.ict.ac.cn:8080/jcst/EN/10.1007/s11390-017-1760-9
Copyright 2010 by Journal of Computer Science and Technology