Abstract: Previous studies demonstrate that word embeddings and part-of-speech (POS) tags are helpful for punctuation restoration tasks. However, two drawbacks still exist. One is that word embeddings are pre-trained by unidirectional language modeling objectives. Thus the word embeddings only contain left-to-right context information. The other is that POS tags are provided by an external POS tagger. So computation cost will be increased and incorrect predicted tags may affect the performance of restoring punctuation marks during decoding. This paper proposes adversarial transfer learning to address these problems. A pre-trained bidirectional encoder representations from transformers (BERT) model is used to initialize a punctuation model. Thus the transferred model parameters carry both left-to-right and right-to-left representations. Furthermore, adversarial multi-task learning is introduced to learn task invariant knowledge for punctuation prediction. We use an extra POS tagging task to help the training of the punctuation predicting task. Adversarial training is utilized to prevent the shared parameters from containing task specific information. We only use the punctuation predicting task to restore marks during decoding stage. Therefore, it will not need extra computation and not introduce incorrect tags from the POS tagger. Experiments are conducted on IWSLT2011 datasets. The results demonstrate that the punctuation predicting models obtain further performance improvement with task invariant knowledge from the POS tagging task. Our best model outperforms the previous state-of-the-art model trained only with lexical features by up to 9.2% absolute overall F_1-score on test set.

Boosting Punctuation Restoration with Data Generation and Reinforcement Learning

Incorporating External POS Tagger for Punctuation Restoration

Punctuation Restoration Improves Structure Understanding without Supervision

Punctuation Restoration for Speech Transcripts using seq2seq Transformers

Punctuation Restoration for Singaporean Spoken Languages: English, Malay, and Mandarin

Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Network

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

Adversarial Transfer Learning for Punctuation Restoration

Zephyr: Zero-Shot Punctuation Restoration

PDAugment: Data Augmentation by Pitch and Duration Adjustments for Automatic Lyrics Transcription

Fast and Accurate Capitalization and Punctuation for Automatic Speech Recognition Using Transformer and Chunk Merging

Resolving Transcription Ambiguity in Spanish: A Hybrid Acoustic-Lexical System for Punctuation Restoration

LLaMA based Punctuation Restoration With Forward Pass Only Decoding

Punctuation Prediction for Streaming On-Device Speech Recognition

Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation

Spontaneous Informal Speech Dataset for Punctuation Restoration

Multimodal Punctuation Prediction with Contextual Dropout

Automatic punctuation generation for speech

A Small and Fast BERT for Chinese Medical Punctuation Restoration

A Linguistically Inspired Statistical Model for Chinese Punctuation Generation

Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition