Abstract:We present MARS (Multilingual Automatic tRanslation System), a research prototype speech-to-speech translation system. MARS is aimed at two-way conversational spoken language translation between English and Mandarin Chinese for limited domains, such as air travel reservations. In MARS, machine translation is embedded within a complex speech processing task, and the translation performance is highly effected by the performance of other components, such as the recognizer and semantic parser, etc. All components in the proposed system are statistically trained using an appropriate training corpus. The speech signal is first recognized by an automatic speech recognizer (ASR). Next, the ASR-transcribed text is analyzed by a semantic parser, which uses a statistical decision-tree model that does not require hand-crafted grammars or rules. Furthermore, the parser provides semantic information that helps further re-scoring of the speech recognition hypotheses. The semantic content extracted by the parser is formatted into a language-independent tree structure, which is used for an interlingua based translation. A Maximum Entropy based sentence-level natural language generation (NLG) approach is used to generate sentences in the target language from the semantic tree representations. Finally, the generated target sentence is synthesized into speech by a speech synthesizer. Many new features and innovations have been incorporated into MARS: the translation is based on understanding the meaning of the sentence; the semantic parser uses a statistical model and is trained from a semantically annotated corpus; the output of the semantic parser is used to select a more specific language model to refine the speech recognition performance; the NLG component uses a statistical model and is also trained from the same annotated corpus. These features give MARS the advantages of robustness to speech disfluencies and recognition errors, tighter integration of semantic information into speech recognition, and portability to new languages and domains. These advantages are verified by our experimental results.

IBM Mastor: Multilingual Automatic Speech-To-Speech Translator

IBM MASTOR System

Handheld Speech to Speech Translation System

Recent Advances of IBM’s Handheld Speech Translation System

Recent Advances of IBM ’ s Handheld

The IBM Speech-to-speech Translation System for Smartphone: Improvements for Resource-Constrained Tasks.

A Hand-Held Speech-to-speech Translation System

Two-way speech-to-speech translation on handheld devices

Design of a speech interaction system for man-machine confrontation

MARS: A Statistical Semantic Parsing and Generation-Based Multilingual Automatic Translation System

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

Multilingual Speech-to-Speech Translation into Multiple Target Languages

A New Mmwave-Speech Multimodal Speech System for Voice User Interface

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Towards High Performance LVCSR in Speech-to-Speech Translation System on Smart Phones.

SimulTron: On-Device Simultaneous Speech to Speech Translation

MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation

PolyVoice: Language Models for Speech to Speech Translation

Verbmobil: Foundations of Speech-to-Speech Translation

Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech

Advances in Syntax-Based Malay-English Speech Translation