Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Baijun Ji; Tong Zhang; Yicheng Zou; Bojie Hu; Si Shen

doi:10.18653/v1/2022.emnlp-main.453

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Baijun Ji, Tong Zhang, Yicheng Zou, Bojie Hu^*, Si Shen

^*Corresponding author for this work

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

5 Citations (Scopus)

Abstract

Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.

Original language	English
Title of host publication	Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022
Editors	Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Publisher	Association for Computational Linguistics (ACL)
Pages	6755-6764
Number of pages	10
ISBN (Electronic)	9781959429401
DOIs	https://doi.org/10.18653/v1/2022.emnlp-main.453
Publication status	Published - 2022
Externally published	Yes
Event	2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 - Hybrid, Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Publication series

Name	Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

Conference

Conference	2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022
Country/Territory	United Arab Emirates
City	Hybrid, Abu Dhabi
Period	7/12/22 → 11/12/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Access to Document

10.18653/v1/2022.emnlp-main.453

Cite this

Ji, B., Zhang, T., Zou, Y., Hu, B., & Shen, S. (2022). Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 (pp. 6755-6764). (Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.emnlp-main.453

Ji, Baijun ; Zhang, Tong ; Zou, Yicheng et al. / Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022. editor / Yoav Goldberg ; Zornitsa Kozareva ; Yue Zhang. Association for Computational Linguistics (ACL), 2022. pp. 6755-6764 (Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022).

@inproceedings{890257062adb4536a2c1c9b939e31b4b,

title = "Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective",

abstract = "Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.",

author = "Baijun Ji and Tong Zhang and Yicheng Zou and Bojie Hu and Si Shen",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

doi = "10.18653/v1/2022.emnlp-main.453",

language = "English",

series = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022",

publisher = "Association for Computational Linguistics (ACL)",

pages = "6755--6764",

editor = "Yoav Goldberg and Zornitsa Kozareva and Yue Zhang",

booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022",

}

Ji, B, Zhang, T, Zou, Y, Hu, B & Shen, S 2022, Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective. in Y Goldberg, Z Kozareva & Y Zhang (eds), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Association for Computational Linguistics (ACL), pp. 6755-6764, 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Hybrid, Abu Dhabi, United Arab Emirates, 7/12/22. https://doi.org/10.18653/v1/2022.emnlp-main.453

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective. / Ji, Baijun; Zhang, Tong; Zou, Yicheng et al.
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022. ed. / Yoav Goldberg; Zornitsa Kozareva; Yue Zhang. Association for Computational Linguistics (ACL), 2022. p. 6755-6764 (Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022).

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

TY - GEN

T1 - Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

AU - Ji, Baijun

AU - Zhang, Tong

AU - Zou, Yicheng

AU - Hu, Bojie

AU - Shen, Si

PY - 2022

Y1 - 2022

N2 - Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.

AB - Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.

UR - http://www.scopus.com/inward/record.url?scp=85149435559&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.emnlp-main.453

DO - 10.18653/v1/2022.emnlp-main.453

M3 - Conference Paper published in a book

AN - SCOPUS:85149435559

T3 - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

SP - 6755

EP - 6764

BT - Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

A2 - Goldberg, Yoav

A2 - Kozareva, Zornitsa

A2 - Zhang, Yue

PB - Association for Computational Linguistics (ACL)

T2 - 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

Y2 - 7 December 2022 through 11 December 2022

ER -

Ji B, Zhang T, Zou Y, Hu B, Shen S. Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective. In Goldberg Y, Kozareva Z, Zhang Y, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022. Association for Computational Linguistics (ACL). 2022. p. 6755-6764. (Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022). doi: 10.18653/v1/2022.emnlp-main.453

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this