Can Informal Genres Be Better Translated by Tuning on Automatic Semantic Metrics?

Even though the informal language of spoken text and web forum genres presents great difficulties for automatic semantic role labeling, we show that surprisingly, tuning statistical machine translation against the SRL-based objective function, MEANT, nevertheless leads more robustly to adequate translations of these informal genres than tuning against BLEU or TER. The accuracy of automatic semantic parsing has been shown to degrade significantly on informal genres such as speech or tweets, compared to formal genres like newswire. In spite of this, human evaluators preferred translations from MEANT-tuned systems over the BLEU- or TER-tuned ones by a significant margin. Error analysis indicates that one of the major sources of errors in automatic shallow semantic parsing of informal genres is failure to identify the semantic frame for copula or existential senses of ``be''. We show that MEANT's correlation with human adequacy judgment on informal text is improved by reconstructing the missing semantic frames for ``be''. Our tuning approach is independent of the translation model architecture, so any SMT model can potentially benefit from the semantic knowledge incorporated through our approach.