Using bidirectional LSTM with BERT for Chinese punctuation prediction

Mingfeng Fang; Haifeng Zhao; Xiao Song; Xin Wang; Shilei Huang

doi:10.1109/ICSIDP47821.2019.9172986

Using bidirectional LSTM with BERT for Chinese punctuation prediction

Mingfeng Fang, Haifeng Zhao, Xiao Song, Xin Wang, Shilei Huang

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

8 Citations (Scopus)

Abstract

Punctuation prediction is an important step in the post processing of ASR systems. Lack of punctuation text is usually difficult to read and understand. In this paper, we propose a method based on Chinese punctuation prediction by combining the Bidirectional Long Short-Term Memory (BLSTM) and the Bidirectional Encoder Representations from Transformers (BERT), which makes the use of BERT as text encoding layers for learning contextualized word representations for improving the performance of BLSTM network. Compared with the previous punctuation prediction methods based on Recurrent Neural Network (RNN), our method improves the performance of punctuation prediction with the powerful ability of capturing semantics and long-distance dependencies in Chinese unsegmented text. Our experimental results on Chinese news datasets that our BERT-BLSTM based method outperforms the baseline by up to 31.07% absolute in overall micro-F1.

Original language	English
Title of host publication	ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728123455
DOIs	https://doi.org/10.1109/ICSIDP47821.2019.9172986
Publication status	Published - Dec 2019
Externally published	Yes
Event	2019 IEEE International Conference on Signal, Information and Data Processing, ICSIDP 2019 - Chongqing, China Duration: 11 Dec 2019 → 13 Dec 2019

Publication series

Name	ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019

Conference

Conference	2019 IEEE International Conference on Signal, Information and Data Processing, ICSIDP 2019
Country/Territory	China
City	Chongqing
Period	11/12/19 → 13/12/19

Bibliographical note

Publisher Copyright:
© 2019 IEEE.

Keywords

neural network
pre-trained language model
punctuation prediction

Access to Document

10.1109/ICSIDP47821.2019.9172986

Cite this

Fang, M., Zhao, H., Song, X., Wang, X., & Huang, S. (2019). Using bidirectional LSTM with BERT for Chinese punctuation prediction. In ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019 Article 9172986 (ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICSIDP47821.2019.9172986

Fang, Mingfeng ; Zhao, Haifeng ; Song, Xiao et al. / Using bidirectional LSTM with BERT for Chinese punctuation prediction. ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019. Institute of Electrical and Electronics Engineers Inc., 2019. (ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019).

@inproceedings{00a7df8e02d243e296c02e1bb059888d,

title = "Using bidirectional LSTM with BERT for Chinese punctuation prediction",

abstract = "Punctuation prediction is an important step in the post processing of ASR systems. Lack of punctuation text is usually difficult to read and understand. In this paper, we propose a method based on Chinese punctuation prediction by combining the Bidirectional Long Short-Term Memory (BLSTM) and the Bidirectional Encoder Representations from Transformers (BERT), which makes the use of BERT as text encoding layers for learning contextualized word representations for improving the performance of BLSTM network. Compared with the previous punctuation prediction methods based on Recurrent Neural Network (RNN), our method improves the performance of punctuation prediction with the powerful ability of capturing semantics and long-distance dependencies in Chinese unsegmented text. Our experimental results on Chinese news datasets that our BERT-BLSTM based method outperforms the baseline by up to 31.07\% absolute in overall micro-F1.",

keywords = "neural network, pre-trained language model, punctuation prediction",

author = "Mingfeng Fang and Haifeng Zhao and Xiao Song and Xin Wang and Shilei Huang",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 2019 IEEE International Conference on Signal, Information and Data Processing, ICSIDP 2019 ; Conference date: 11-12-2019 Through 13-12-2019",

year = "2019",

month = dec,

doi = "10.1109/ICSIDP47821.2019.9172986",

language = "English",

series = "ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019",

}

Fang, M, Zhao, H, Song, X, Wang, X & Huang, S 2019, Using bidirectional LSTM with BERT for Chinese punctuation prediction. in ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019., 9172986, ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019, Institute of Electrical and Electronics Engineers Inc., 2019 IEEE International Conference on Signal, Information and Data Processing, ICSIDP 2019, Chongqing, China, 11/12/19. https://doi.org/10.1109/ICSIDP47821.2019.9172986

Using bidirectional LSTM with BERT for Chinese punctuation prediction. / Fang, Mingfeng; Zhao, Haifeng; Song, Xiao et al.
ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019. Institute of Electrical and Electronics Engineers Inc., 2019. 9172986 (ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019).

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

TY - GEN

T1 - Using bidirectional LSTM with BERT for Chinese punctuation prediction

AU - Fang, Mingfeng

AU - Zhao, Haifeng

AU - Song, Xiao

AU - Wang, Xin

AU - Huang, Shilei

PY - 2019/12

Y1 - 2019/12

N2 - Punctuation prediction is an important step in the post processing of ASR systems. Lack of punctuation text is usually difficult to read and understand. In this paper, we propose a method based on Chinese punctuation prediction by combining the Bidirectional Long Short-Term Memory (BLSTM) and the Bidirectional Encoder Representations from Transformers (BERT), which makes the use of BERT as text encoding layers for learning contextualized word representations for improving the performance of BLSTM network. Compared with the previous punctuation prediction methods based on Recurrent Neural Network (RNN), our method improves the performance of punctuation prediction with the powerful ability of capturing semantics and long-distance dependencies in Chinese unsegmented text. Our experimental results on Chinese news datasets that our BERT-BLSTM based method outperforms the baseline by up to 31.07% absolute in overall micro-F1.

AB - Punctuation prediction is an important step in the post processing of ASR systems. Lack of punctuation text is usually difficult to read and understand. In this paper, we propose a method based on Chinese punctuation prediction by combining the Bidirectional Long Short-Term Memory (BLSTM) and the Bidirectional Encoder Representations from Transformers (BERT), which makes the use of BERT as text encoding layers for learning contextualized word representations for improving the performance of BLSTM network. Compared with the previous punctuation prediction methods based on Recurrent Neural Network (RNN), our method improves the performance of punctuation prediction with the powerful ability of capturing semantics and long-distance dependencies in Chinese unsegmented text. Our experimental results on Chinese news datasets that our BERT-BLSTM based method outperforms the baseline by up to 31.07% absolute in overall micro-F1.

KW - neural network

KW - pre-trained language model

KW - punctuation prediction

UR - https://www.scopus.com/pages/publications/85091918498

U2 - 10.1109/ICSIDP47821.2019.9172986

DO - 10.1109/ICSIDP47821.2019.9172986

M3 - Conference Paper published in a book

AN - SCOPUS:85091918498

T3 - ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019

BT - ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2019 IEEE International Conference on Signal, Information and Data Processing, ICSIDP 2019

Y2 - 11 December 2019 through 13 December 2019

ER -

Fang M, Zhao H, Song X, Wang X, Huang S. Using bidirectional LSTM with BERT for Chinese punctuation prediction. In ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 9172986. (ICSIDP 2019 - IEEE International Conference on Signal, Information and Data Processing 2019). doi: 10.1109/ICSIDP47821.2019.9172986

Using bidirectional LSTM with BERT for Chinese punctuation prediction

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this