End-to-End Speech-to-Text Translation: A Survey
端到端语音到文本翻译研究综述

Speech data can be augmented in various ways. For example, by adding noise, speed and pitch perturbation, time and frequency masking to name a few. SpeechAugment (Park et al., 2019) policy consists of warping the features, masking blocks of frequency channels, and time steps. It has been successfully used both for ASR (Vincent et al., 2017) and ST tasks (Bahar et al., 2019b). MixSpeech (Meng et al., 2021) as shown in Fig. 4(a) takes the weighted combination of two different speech features as input and two recognition losses with the same weights. A generalization of MixSpeech (Xie and Hansen, 2023) called MixRep applies the mixup idea to the acoustic feature and hidden layers inputs. MixRep combination with a regularization term along the time axis further improves ASR performance. Both MixSpeech and MixRep have been shown to perform well for low-resource ASR and their effectiveness is still to be tested for ST tasks. M3ST (Cheng et al., 2022) applies two levels of Fine-Tuning (FT) using mixup data– word, sentence, and frame level mix data in the first FT level and source speech and transcription mixup in the second FT level. M3ST achieves SOTA on MuST-C compared to baselines.
语音数据可通过多种方式进行增强，例如添加噪声、速度和音高扰动、时频掩蔽等。SpeechAugment（Park 等人，2019）策略包含特征扭曲、频率通道块掩蔽和时间步掩蔽，已成功应用于自动语音识别（Vincent 等人，2017）和语音翻译任务（Bahar 等人，2019b）。如图 4(a)所示，MixSpeech（Meng 等人，2021）采用两种不同语音特征的加权组合作为输入，并施加相同权重的双重识别损失。其泛化版本 MixRep（Xie 和 Hansen，2023）将混合增强思想应用于声学特征和隐藏层输入，结合时间轴正则化项进一步提升了语音识别性能。MixSpeech 与 MixRep 在低资源语音识别中表现优异，但其在语音翻译任务中的有效性仍有待验证。M3ST（Cheng 等人，2022）采用两级微调：第一级使用词汇/句子/帧级混合数据，第二级采用源语音与转写文本的混合增强。在 MuST-C 基准测试中，M3ST 取得了当前最优性能。

It is possible to augment both speech and text simultaneously and create new paired data. For example, sample, translate, and recombine (Lam et al., 2022b) first samples a suffix replacement from suffix memory corresponding to a pivot token from transcription. It then translates the combined new utterance (prefix+pivot+replacmenet suffix) to generate a new target sentence. The corresponding audio pair is obtained by concatenating the audio frames of the prefix, pivot, and replacement suffix. The interesting thing about the proposed method is that it can generate real-looking sentences contrary to pseudo-sentences. Concatenation of original ST data has been used to augment the entire training data (Lam et al., 2022a). In particular, (Lam et al., 2022a) proposes CatSpeaker that uses single speaker information and CatRandom that randomly generates audio-text pairs spoken by different speakers.
可以同时对语音和文本进行增强并创建新的配对数据。例如，采样-翻译-重组方法（Lam 等人，2022b）首先从后缀存储器中采样与转录文本中的枢轴标记相对应的后缀替换项，随后将组合后的新话语（前缀+枢轴+替换后缀）翻译生成新的目标语句。对应的音频对则是通过拼接前缀、枢轴及替换后缀的音频帧获得。该方法的有趣之处在于能生成真实感语句而非伪句子。原始语音翻译数据的串联拼接已被用于增强整个训练集（Lam 等人，2022a）。具体而言，（Lam 等人，2022a）提出了使用单一说话者信息的 CatSpeaker 方案，以及随机生成不同说话者音频-文本对的 CatRandom 方案。

ST models often use ASR transcripts and MT translations as auxiliary data which needs to be fed to the encoder and decoder, respectively. To learn representation for such text data, existing works rely on word embedding (Zhang et al., 2023c; Bérard et al., 2016), LSTM (Kim et al., 2017; Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019), and Transformer (Wang et al., 2021b; Liu et al., 2021a; Zeng et al., 2021). Text data is often tokenized and fed as either a word or as a character (Bérard et al., 2018). The output of the decoder could be graphene, characters, or words.
语音转文本模型常将自动语音识别(ASR)转录文本和机器翻译(MT)译文作为辅助数据，分别输入编码器和解码器。为学习此类文本数据的表征，现有研究主要采用词嵌入(Zhang et al., 2023c; Bérard et al., 2016)、长短期记忆网络(Kim et al., 2017; Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019)以及 Transformer 架构(Wang et al., 2021b; Liu et al., 2021a; Zeng et al., 2021)。文本数据通常经过分词处理后，以单词或字符形式输入(Bérard et al., 2018)。解码器输出可以是字形、字符或单词。

ST models take speech as input and utilize various speech-based feature representation methods to convert speech into a vector representation. Traditional speech feature extraction methods such as Perceptual Linear Prediction, (PLP), Fbank, and Mel-Filter Cepstral Coefficient (MFCC) (Rabiner and Schafer, 2010) have been used after normalization to extract speech features by many (Duong et al., 2016; Bérard et al., 2016; Kim et al., 2017; Bérard et al., 2018; Anastasopoulos and Chiang, 2018; Bansal et al., 2019; Jia et al., 2019; Inaguma et al., 2019; Liu et al., 2020d; Dong et al., 2021; Le et al., 2023b; Parcollet et al., 2024), sometimes combining them with pitch features and speech augmentation methods as described in §5. These feature extraction methods are sometimes being replaced by distributed feature representation methods such as speech word2vec (Chung and Glass, 2018) owing to their dense continuous feature representation capability.
语音到文本（ST）模型以语音作为输入，采用多种基于语音的特征表示方法将语音转换为向量表征。传统语音特征提取方法如感知线性预测（PLP）、滤波器组（Fbank）和梅尔频率倒谱系数（MFCC）（Rabiner and Schafer, 2010）经过归一化处理后，已被众多研究者（Duong et al., 2016; Bérard et al., 2016; Kim et al., 2017; Bérard et al., 2018; Anastasopoulos and Chiang, 2018; Bansal et al., 2019; Jia et al., 2019; Inaguma et al., 2019; Liu et al., 2020d; Dong et al., 2021; Le et al., 2023b; Parcollet et al., 2024）用于语音特征提取，有时会如第 5 节所述结合基频特征与语音增强方法使用。由于具备稠密连续特征表示能力，这些特征提取方法正逐渐被语音 word2vec（Chung and Glass, 2018）等分布式特征表示方法所替代。

It is difficult to get a large amount of labeled speech data to learn supervised speech feature representation. Therefore, more recent works exploit speech features learned via unsupervised and self-supervised ways, mapping continuous speech signal to discrete units– akin to words and sub-words in the text domain. Such a representation facilitates tools developed in NLP to borrow in the speech domain. Among them the most popular is Wav2Vec (Schneider et al., 2019) and its variants such as w2v-BERT (Chung et al., 2021) and Wav2vec 2.0 (Baevski et al., 2020) used in (Tran et al., 2020; Le et al., 2020; Li et al., 2020; Han et al., 2021; Popuri et al., 2022; Zhang et al., 2023c). Interestingly, Wav2Vec and its variants can be used as an encoder in a Seq2Seq framework alone or combined with adapters and CNN for Length shrinking³³3Length shrinking is an important issue in ST task since speech is a much longer sequence than text. Therefore, existing works employ various techniques such as length adapters, CNN, CTC for length shrinking. A few works such as CSTNet (Khurana et al., 2020; Wang et al., 2020d) use CNN for feature extraction and length shrinking.
获取大量标注语音数据以学习有监督的语音特征表示存在困难。因此，近期研究多采用无监督和自监督方式学习语音特征，将连续语音信号映射为离散单元——类似于文本领域的单词和子词。这种表征方式便于将自然语言处理领域的工具迁移至语音领域。其中最流行的是 Wav2Vec（Schneider 等人，2019）及其变体，如 w2v-BERT（Chung 等人，2021）和 Wav2vec 2.0（Baevski 等人，2020），这些模型已被应用于（Tran 等人，2020；Le 等人，2020；Li 等人，2020；Han 等人，2021；Popuri 等人，2022；Zhang 等人，2023c）等研究。值得注意的是，Wav2Vec 及其变体可单独作为 Seq2Seq 框架中的编码器，也可与适配器和 CNN 结合用于长度缩减 ³ 。部分研究如 CSTNet（Khurana 等人，2020；Wang 等人，2020d）则采用 CNN 进行特征提取和长度缩减。

More recent works in ST are employing HuBERT (Hsu et al., 2021) for speech representation (among other benefits of HuBERT) (Zhang et al., 2023a). Hubert offers stable training and better targets than Wav2Vec 2.0 since it uses hidden layers representation during the clustering process. For encoding long-speech signals, Conformers (Gulati et al., 2020) can be used as they provide local context through convolution block and global context through an attention mechanism. Seamless4MT (Barrault et al., 2023) uses conformer for speech encoding.
近期语音翻译研究开始采用 HuBERT（Hsu 等人，2021）作为语音表征方法（Zhang 等人，2023a），这得益于 HuBERT 的多重优势。相较于 Wav2Vec 2.0，HuBERT 在聚类过程中采用隐藏层表征，能提供更稳定的训练目标和更优的性能指标。针对长语音信号编码，可选用 Conformer 模型（Gulati 等人，2020），其通过卷积模块捕获局部上下文，并借助注意力机制获取全局上下文。Seamless4MT 系统（Barrault 等人，2023）即采用 Conformer 架构进行语音编码。

Other speech representation techniques such as VQ-VAE (van den Oord et al., 2017), WavLM (Chen et al., 2022), data2vec (Baevski et al., 2022), Robust data2vec (Zhu et al., 2023), SpeechLM (Zhang et al., 2024b), may also be explored while encoding speech for ST tasks.
其他语音表征技术，如 VQ-VAE（van den Oord 等，2017）、WavLM（Chen 等，2022）、data2vec（Baevski 等，2022）、Robust data2vec（Zhu 等，2023）、SpeechLM（Zhang 等，2024b）等，在语音到文本翻译任务的编码过程中同样值得探索。

The speech and text in an ST task are semantically related because both of them refer to the same thing. Therefore, it is imperative to learn a joint speech-text representation in the hope of bridging the modality gap between them. A method for learning a combined representation of text and speech is called modality bridging (see fig.5). Hence, a good ST model should learn a representation such that embeddings of both modalities for similar speech-text pairs lie close to each other. It is believed that low performance on ST tasks is due to models not learning aligned representations of speech and text. Therefore, different authors have devised different ways to fill the gap, which fall into five major approaches: (a) adapters, (b) contrastive learning, (c) knowledge-distillation, (d) optimal transport, and (e) mix-up strategy. Below we discuss the works utilizing these approaches and show the pros and cons.
在语音到文本（ST）任务中，语音与文本具有语义关联性，因为二者指向同一事物。因此，学习联合语音-文本表征以弥合模态差异至关重要。这种学习语音与文本联合表征的方法称为模态桥接（见图 5）。理想的 ST 模型应习得一种表征方式，使得相似语音-文本对的两种模态嵌入在空间中彼此接近。当前 ST 任务性能不佳的主要原因被认为是模型未能学习到语音与文本的对齐表征。为此，不同研究者提出了填补这一差距的五类主要方法：(a)适配器、(b)对比学习、(c)知识蒸馏、(d)最优传输、(e)混合策略。下文将分别探讨采用这些方法的研究成果，并分析其优劣。

1. 1.

   Adapters are small modules integrated with pre-trained networks for specific tasks (Houlsby et al., 2019). They have performed on par with fine-turning-based approaches while requiring only a fraction of trainable parameters. For example, in (Gállego et al., 2021; Zhao et al., 2022; Sarkar et al., 2023), the modality gap is filled using adapter layers, which is a multi-headed self-attention with pooling operation. The author uses Wav2Vec 2.0 (Baevski et al., 2020) for speech-feature extraction, wherein self-attention layers in the transformer are equipped with pooling operation for dimensionality reduction to match the text representation.

   Report issue for preceding element
   
   1. 适配器是与预训练网络集成的小型模块，用于特定任务（Houlsby 等人，2019）。其性能与基于微调的方法相当，但仅需少量可训练参数。例如在（Gállego 等人，2021；Zhao 等人，2022；Sarkar 等人，2023）中，研究者使用带池化操作的多头自注意力适配器层来填补模态鸿沟。作者采用 Wav2Vec 2.0（Baevski 等人，2020）进行语音特征提取，其中变压器的自注意力层配备池化操作以实现降维，从而匹配文本表征。
2. 2.

   Contrastive learning approximates the “semantic” distance in the input space using a simple distance in the target space after mapping input patterns onto the target space (Chopra et al., 2005). It tries to bring positive instances closer while pushing negative ones apart. It has been used excessively in both supervised and unsupervised settings for learning representations. For example, (Zhang et al., 2023c) performs the explicit knowledge transfer through contrastive learning. It learns frame and sentence-level speech feature representation and uses whitening (Su et al., 2021) to alleviate the MT representation degeneration. (Liu et al., 2019) decouples the encoder representation into three parts: acoustic encoder, shrinking (done via CTC) of acoustic encoder output, and semantic encoder for modality-gap bridging. Using a contrastive learning architecture, Chimera (Han et al., 2021) trains a semantic memory module which is shared for overcoming the modality distance. XSTNet (Ye et al., 2021) augmented with contrastive loss (Ye et al., 2022a) investigates three different methods: span masked representation, word-repetition and cut-off. It claims that contrastive loss is better than CTC and L2 loss. Word-aligned contrastive learning (WACO) (Ouyang et al., 2023) bridges the modality gap by forming average speech and word embedding of the same word as the positive pair while of different words as negative pairs. CSTNet is a self-supervised learning framework based on contrastive learning (using a mix of triplet losses)(Khurana et al., 2020). On top of the CTC loss, the boundary-based speech length shrinking mechanism is applied in (Zeng et al., 2022). The authors claim that if boundary-based shrinking is applied with other modality-bridging techniques, such as contrastive loss, it can further improve the model performance. The approach presented achieves lower inference speed and memory footprint. (Yin et al., 2023) proposes a novel integration of speech and text, referred to as a third modality. This fusion is achieved through the application of Cross-modal Contrastive Learning (Sohn, 2016) and Cross-Attentive Regularization (Tang et al., 2021a). Additionally, the method incorporates techniques such as Knowledge Distillation and Jensen-Shannon Divergence (Lin, 1991; Liu et al., 2019; Gaido et al., 2020a) to bridge the modality gap, addressing challenges related to input representation, semantics, and hidden states.
   对比学习通过将输入模式映射到目标空间后，利用目标空间中的简单距离来近似输入空间的"语义"距离（Chopra 等，2005）。该方法致力于拉近正样本间距同时推远负样本，已在监督与非监督场景中被广泛用于表征学习。例如（Zhang 等，2023c）通过对比学习实现显式知识迁移，其同时学习帧级和句子级语音特征表示，并采用白化处理（Su 等，2021）缓解机器翻译表征退化问题。（Liu 等，2019）将编码器表征解耦为三部分：声学编码器、声学编码输出的 CTC 收缩模块以及用于模态间隙桥接的语义编码器。Chimera（Han 等，2021）采用对比学习架构训练共享语义记忆模块以克服模态距离。增强对比损失的 XSTNet（Ye 等，2021；Ye 等，2022a）探索了三种方法：跨度掩码表示、词语重复与截断策略。 研究表明对比损失函数优于 CTC 和 L2 损失函数。词对齐对比学习（WACO）（Ouyang 等，2023）通过将同一词语的语音嵌入向量与文本嵌入向量的均值构成正样本对，不同词语的嵌入向量构成负样本对，从而弥合模态差异。CSTNet 是基于对比学习的自监督框架（采用混合三元组损失函数）（Khurana 等，2020）。在 CTC 损失函数基础上，（Zeng 等，2022）应用了基于边界的语音长度压缩机制。作者指出，若将基于边界的压缩机制与其他模态桥接技术（如对比损失）结合使用，可进一步提升模型性能。该方法实现了更低的推理时延和内存占用。（Yin 等，2023）提出了一种称为"第三模态"的语音文本融合新范式，通过跨模态对比学习（Sohn，2016）和交叉注意力正则化（Tang 等，2021a）实现模态融合。 此外，该方法融合了知识蒸馏与 Jensen-Shannon 散度（Lin, 1991；Liu 等, 2019；Gaido 等, 2020a）等技术以弥合模态差异，解决了输入表征、语义及隐状态相关的挑战。

   Report issue for preceding element

   Models/Techniques  模型/技术	Problem Solved  问题解决	Dataset  数据集	Language Pair  语言对	Speech (hours)  语音时长（小时）	Metric (BLEU)  度量标准（BLEU）
   M-Adapter + W2V2 + mBart (Baevski et al., 2020)	training gap between Pre-training & Fine-tuning the modality	MuST-C	En$\rightarrow$De	408	25.9
   			En$\rightarrow$Ro	432	24.62
   			En$\rightarrow$Fr	492	37.34
   Chimera (Han et al., 2021)	projecting audio & text to a common semantic representation 将音频与文本投影至共同的语义表征空间	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	408	27.1
   			En$\rightarrow$Fr  英译法	492	35.6
   			En$\rightarrow$Ru  恩 $\rightarrow$ 俄	489	17.4
   			En$\rightarrow$Es  英语 $\rightarrow$ 西班牙语	504	30.6
   			En$\rightarrow$It  英语 $\rightarrow$ 意大利语	465	25.0
   			En$\rightarrow$Ro  英语 $\rightarrow$ 罗马尼亚语	432	24.0
   			En$\rightarrow$Pt  恩 $\rightarrow$ 葡	385	30.2
   			En$\rightarrow$Nl  英语 $\rightarrow$ 荷兰语	442	29.2
   ConST (XSTNet + Constrastive Loss) (Ye et al., 2021) ConST（XSTNet + 对比损失） 叶等人 2021	closes modality gap  消除模态差距	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	408	28.3
   			En$\rightarrow$Es  英语 $\rightarrow$ 西班牙语	504	32.0
   			En$\rightarrow$Fr  英译法	492	38.3
   			En$\rightarrow$It  英语 $\rightarrow$ 意大利语	465	27.2
   			En$\rightarrow$Nl  英语 $\rightarrow$ 荷兰语	442	31.7
   			En$\rightarrow$Pt  恩 $\rightarrow$ 葡	385	33.1
   			En$\rightarrow$Ro  英语 $\rightarrow$ 罗马尼亚语	432	25.6
   			En$\rightarrow$Ru  恩 $\rightarrow$ 俄	489	18.9
   W2V2 + mBart + Adapter (Gállego et al., 2021; Zhao et al., 2022) W2V2 + mBart + 适配器（Gállego 等，2021；赵等，2022）	slow convergence speed  收敛速度慢	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	408	28.22
   WACO (Ouyang et al., 2023) WACO Ouyang 等人，2023	limited parallel data (1-hour) 有限的平行数据（1 小时）	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	1	17.5
   AdaTrans (Zeng et al., 2022) AdaTrans Zeng 等人，2022	closing gap between length of speech & text 缩小语音与文本长度差距	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	408	28.7
   			En$\rightarrow$Fr  英译法	492	38.7
   			En$\rightarrow$Ru  恩 $\rightarrow$ 俄	489	19.0
   STEMM (Fang et al., 2022) 方等人，2022 年提出的 STEMM	Speech representation  语音表征	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	408	28.7
   			En$\rightarrow$Fr  英译法	492	37.4
   			En$\rightarrow$Ru  恩 $\rightarrow$ 俄	489	17.8
   			En$\rightarrow$Es  英语 $\rightarrow$ 西班牙语	504	31.0
   			En$\rightarrow$It  英语 $\rightarrow$ 意大利语	465	25.8
   			En$\rightarrow$Ro  英语 $\rightarrow$ 罗马尼亚语	432	24.5
   			En$\rightarrow$Pt  恩 $\rightarrow$ 葡	385	31.7
   			En$\rightarrow$Nl  英语 $\rightarrow$ 荷兰语	442	30.5
   CTC loss + Optimal Transport (Siamese-PT) (Le et al., 2023b) CTC 损失函数 + 最优传输（Siamese-PT）Le 等人，2023b	without change in architecture 无需改变架构	MuST-C  MuST-C 语料库	En$\rightarrow$De  英译德	408	27.9
   			En$\rightarrow$Es	504	31.8
   			En$\rightarrow$Fr	492	39.2
   			En$\rightarrow$It	465	27.7
   			En$\rightarrow$Nl	442	31.7
   			En$\rightarrow$Pt	385	34.2
   			En$\rightarrow$Ro	432	27.0
   			En$\rightarrow$Ru	489	18.5
   Fine & Coarse Granularity Contrastive Learning (Zhang et al., 2023c)	limited knowledge transfer ability	MuST-C	En$\rightarrow$De	408	29.0
   			En$\rightarrow$Fr	492	38.3
   			En$\rightarrow$Ru	489	19.7
   			En$\rightarrow$Es	504	31.9
   			En$\rightarrow$It	465	27.3
   			En$\rightarrow$Ro	432	26.8
   			En$\rightarrow$Pt	385	32.7
   			En$\rightarrow$Nl	442	31.6

   Report issue for preceding element
3. 3.

   Knowledge-distillation (Hinton et al., 2015) is a mechanism to distill information from a trained and large “teacher” model to a smaller and efficient “student” model. It has been used with $L_{2}$ loss in (Huzaifah and Kukanov, 2023) to address the modality gap issue.

   Report issue for preceding element
4. 4.

   Optimal transport (OT) (Peyré et al., 2019) is a mechanism for comparing two probability distributions . In the ST task, speech and text representations may be deemed as two probability distributions, and therefore, OT can be applied. More formally, suppose $\alpha$ and $\beta$ denote the discrete probability distributions corresponding to speech and text representations. The masses at each position $u_{i}$ and $v_{i}$ are $a_{i}$ and $b_{j}$ respectively such that $\sum_{i=1}^{m}a_{i}=1$ and $\sum_{j=1}^{n}b_{j}$. Suppose further that the cost of transporting a unit of mass from $u_{i}$ to $v_{j}$ is $c(u_{i},v_{j})$, where $c$ is some cost function such as Euclidean distance. Let $Z_{ij}\geq 0$ be the quantity of mass to be transported from $u_{i}$ to $v_{j}$ then the goal of OT is to move all masses from $\alpha$ to $\beta$ such that the following objective function is minimized

   Report issue for preceding element

   $$\min_{Z}\langle C,Z\rangle,\qquad Z{\bf 1}_{n}=a,Z^{T}{\bf 1}_{m}=b,Z\geq 0$$

   (21)

   In the above eq., $C$ and $Z$ are the matrices whose elements are $c_{ij}=c(u_{i},v_{j})$ and $Z_{ij}$, respectively. ${\bf 1}$ denotes the vector of ones. In ST task, $c(u_{i},v_{j})=\|u_{i}-v_{j}\|_{p}$ for some $p\geq 1$. The loss corresponding to (21) is called Wassertein loss optimizing which is costly. Hence an entropy-regularized upper-bound approximation is often optimized

   Report issue for preceding element

   $$\min_{Z}\{\langle C,Z\rangle-\lambda H(Z)\}$$

   (22)

   where $\lambda$ is a regularization parameter and $H(\cdot)$ is the von-Neuman entropy matrix.

   Report issue for preceding element

   Recent works make use of the OT as presented above. For example, (Le et al., 2023b) uses optimal transport and CTC together to close the modality gap during the pre-training phase. They show significant gains in BLEU score when the ST model is fine-tuned without any external data compared to multitask learning. Similarly, (Tsiamas et al., 2024, 2023) uses OT+CTC to align the speech-encoder representation space with the MT embedding space whereas (Zhou et al., 2023) aligns the two representations via OT followed by cross-modal mix-up at the token level.

   Report issue for preceding element
5. 5.

   Mix-up strategy: Speech-Text Manifold Mixup (STEMM) (Fang et al., 2022) strategy uses speech embedding. It mixes embeddings of speech and text into the encoder-decoder of a translation model for bridging the modality gap under the self-supervised learning framework. PromptST (Yu et al., 2023) presents a linguistic probing learning strategy, referred to as Speech-Senteval, inspired by the approach introduced by (Conneau et al., 2018). This strategy is implemented on the higher layer of the encoder within pre-trained ST models, specifically targeting the challenges associated with learning linguistic properties that these models often struggle with at the higher layers.

   Report issue for preceding element

   Table 1 presents the performance scores of ST models based on modality-bridging techniques. We can observe that mixup strategy achieves the highest BLEU score on En-De pair. Whereas boundary-based speech length shrinking mechanism matches the score when combined with other modality-bridging techniques.

   Report issue for preceding element

Discussion: The study finds that adapters can shrink the speech length as well as the modality distance between the text and speech representations while requiring a small number of trainable parameters. The contrastive loss is found to be better than CTC and $L_{2}$ loss for modality-bridging. The boundary-based speech length shrinking combined with contrastive loss may improve the ST task performance. Finally, it is possible to build ST models requiring zero parallel ST data (Tsiamas et al., 2024).

Offline frameworks perform ST tasks where output tokens are produced after having seen the entire speech utterance. These frameworks heavily rely on Seq2Seq architecture as shown in Fig. 6. It has an encoder for speech input, a decoder for text output, and an optional shared/semantic decoder connecting the encoder and the decoder. The model is usually optimized for the ST loss or sometimes in a multitask learning framework where ASR/MT/CTC (Graves et al., 2006) losses are combined with ST loss. Other times Transfer learning is utilized for leveraging pre-trained models for ST tasks. Another approach that has been gaining a lot of attention is Non-Autoregressive modeling (NAR) for the E2E ST task which gives faster inference. The following section will delve deeper into these approaches.

The Seq2Seq-based ST models proposed in the literature either use specialized encoders such as transformers or attention mechanisms which we discuss next.

1. 1.

   Attention mechanism is used to concentrate on specific sections of the input data instead of the entire data (Larochelle and Hinton, 2010; Mnih et al., 2014; Vaswani et al., 2017). It has been a successful strategy for getting state-of-the-art (SOTA) results in NLP, computer vision, and other areas. There exist various types of attention in the literature such as soft, hard, local, monotonic, multihead, self- and cross-attention, inter alia. For more details, interested readers are encouraged to skim through (Mnih et al., 2014; Vaswani et al., 2017; Brauwers and Frasincar, 2022). Below we provide efforts made to handle ST tasks using the attention mechanism within the Seq2Seq framework.

   Report issue for preceding element

   The convolutional attention to “remember” and avoid translating the signal twice is used within Seq2Seq by (Bérard et al., 2016), which outperforms a hierarchical encoder with better results on synthetic data without using transcripts. The same author in (Bérard et al., 2018) uses source transcript and achieves results close to cascade models on LibriSpeech data. In (Duong et al., 2016), the author proposes phone-to-text alignment with a structural bias feature in the attention model. The measurement of alignment has been explored in (Anastasopoulos et al., 2016), which uses IBM’s translation model as well as dynamic time warping⁴⁴4dynamic time warping (DTW) is an algorithm for measuring similarity between two temporal sequences, which may vary in speed.. Seq2seq with attention trained using multitask learning achieves promising results in (Weiss et al., 2017). These models, however, struggle with noisy inputs and long acoustic signals (Kim et al., 2017). They use a joint CTC-attention model (Graves et al., 2006) trained through multitask learning by incorporating regularizers. The author uses two decoders where the second decoder seeks higher level representation (HLR) from the first decoder besides the encoder via the attention mechanism. Attention-Passing Model (APM) (Sperber et al., 2019), which only passes high-attention vectors from the audio encoder to the translation text for decoding demands a smaller amount of data for training.

   Report issue for preceding element
2. 2.

   Transformer is the architecture based on multi-headed self-attention (Vaswani et al., 2017) which produces contextualized representation of the input. Because of parallelization and contextual representation, transformers have outperformed RNNs on several NLP tasks. This entails us applying transformers for the ST task as well. Transformer-based Seq2Seq with attention is proposed in (Cattoni et al., 2021). The architecture has a quadratic memory complexity, which involves: (a) CNN to downsample the input, and (b) 2-D attention to address short-range dependencies of spectrograms. In (Alastruey et al., 2022), the weight of some attention is avoided for speech tasks, hence decreasing the size of the attention matrix. The transformer encodes the speech features, thereby introducing local self-attention with a suitable window size for each layer to reduce the computational complexity. Other transformer variants which reduce its quadratic complexity such as perceivers (Jaegle et al., 2021) have been used as an encoder (Tsiamas et al., 2022b). Besides quadratic complexity, transformers require lossy downsampling of speech features thus potentially throwing useful linguistic information. To tackle such issues, Speechformers have been proposed (Papi et al., 2021a) which aggregates information at higher layers based on more informed linguistic criteria.

   Report issue for preceding element

As discussed earlier, multitask learning combines the optimization of ST loss with an auxiliary loss such as ASR/MT/CTC loss. Another direction that has been explored by ST researchers is transfer learning in that Seq2Seq encoder/decoders are first pre-trained using ASR/MT data respectively and then the entire model is fine-tuned using ST data. Below, we discuss works based on multitask/transfer learning frameworks.

1. 1.

   ST with ASR: ST with ASR models make use of the transcript data along with speech-text pairs for pre-training. For example, curriculum pre-training (Wang et al., 2020d) refers to using ASR data for pre-training a Seq2Seq model, allowing it to learn transcription. The author argues that if the model is further pre-trained on learning semantic concepts (via frame-based masked language modeling) and word alignment (via frame-based bilingual lexical translation), it boosts the ST task performance. Specifically, existing E2E models either pre-train the encoder or use multi-task learning for ST tasks. As such, the encoder cannot isolate the learning of three tasks: transcription, semantic concept, and alignment, which are segregated by dividing the labor, and experiments prove the theoretical claims. Listen, Understand, and Translate (LUT) (Dong et al., 2021) uses the Seq2Seq model with external ASR loss. Their primary contribution is to introduce a semantic encoder network, whose task is to use the encoder’s output from transcription to minimize the mean-squared loss between the semantic representations and the BERT embeddings of the target text. Such a strategy implicitly builds and trains an NMT model for translation. Pre-training using ASR and/or MT has also been found useful in low-resource scenarios (Zhang et al., 2022a).

   Report issue for preceding element
2. 2.

   ST using MT: This section discusses approaches that use either MT data for pre-training or directly using a pre-trained MT model in the ST decoder. These approaches rely on the idea of generating pseudotext and then translating them using MT. For example, Unsupervised Term Discovery (UTD) (Bansal et al., 2017) groups repeated words into pseudo-text, which is subsequently used for training an MT model using the parallel pseudo-text and target translations. The main advantage of such a system is that it can translate some content words under low-resource settings. The overall results are not very promising on the Spanish-English Call-Home dataset. Another limitation of this work is that the approach is not an E2E in a true sense as it involves two models– a UTD and an MT model. A weakly supervised learning method for ST (Jia et al., 2019) that outperforms multi-task learning takes advantage of the pre-trained MT and TTS synthesis module. Pre-trained MT model is used as a teacher to guide the student ST model in (Liu et al., 2019) (such an approach is dubbed as knowledge distillation (KD)). They, however, rely on source language text and do not improve upon the pipeline system. Following along, (Gaido et al., 2020b) explores word, sentence and sequence-interpolation based KD approaches for transferring knowledge from pre-trained MT to ST model.

   Report issue for preceding element
3. 3.

   ST using both MT and ASR: This section discusses works employing MT and ASR pre-trained models (Bahar et al., 2020; Tsiamas et al., 2022a) or losses for transfer or multitask learning.

   Report issue for preceding element

   Multitask learning proves to be effective when CTC loss is combined with ASR and MT loss in (Bahar et al., 2019a) using various E2E ST architectures such as direct, multitask many-to-one, one-to-many, tied-cascade, and tied-triangle. They show that pre-trained models with ASR and MT losses achieve promising results. Contrary to claims of (Anastasopoulos and Chiang, 2018), tied-triangle architecture is no better than a direct model when fine-tuned properly. Since the ST task is similar to the MT task from the output perspective, works such as XSTNet (Ye et al., 2021) utilize external MT data to pre-train the encoder-decoder network extensively, then fine-tune it using parallel corpus data of MT, ST, ASR, and external MT data for optimizing the model using what they call progressive training. They achieve impressive performance on MuST-C and augmented Librispeech data. They also demonstrate improved performance on auxiliary tasks of MT and ASR. STPT model (Tang et al., 2022) proposes four sub-tasks for multitask pre-training: text-to-text (T2T), which is self-supervised; speech-to-phoneme which is supervised; acoustic learning, which is self-supervised, and ST which is supervised. Only T2T and ST tasks would subsequently be used for fine-tuning. Despite pre-training on “unlabeled” speech data, they obtained superior results on MuST-C data for the ST task. COSTT (Dong et al., 2020) pre-trains encoder using ASR data, the decoder using paired MT data, and then fine-tunes for the joint transcription-translation task. ComSL is a composite ST model relying on multitask learning with three losses ($L_{ASR},L_{MT},L_{ST}$) combined with cross-modality loss to bridge the gap (Le et al., 2023a). It is worth mentioning that ComSL does not require forced-align ST data and learns the cross-modality alignment during training. This however requires optimizing four different losses, viz. Masked Token Prediction, Speech to Text Mapping, Encoder Representation Matching, and Decoder Distribution Matching⁵⁵5please see (Le et al., 2023a) paper for more details. similar to (Tang et al., 2021b). Fused acoustic and text encoding-ST (FAT-ST) (Zheng et al., 2021b) follows the similar pre-training and fine-tuning idea as ComSL except that they propose to use any combination of training data from $D_{2^{\{u,x,v\}}}$ ⁶⁶6$2^{\{u,x,v\}}$ is the power set of triplets.. Essentially, they rely on masked language modeling (MLM) and translation language modeling (TLM) for pre-training (Conneau and Lample, 2019).

   Report issue for preceding element
4. 4.

   Non-Autoregressive Modeling⁷⁷7 We present the discussion of NAR within the multitask learning framework because all NAR E2E ST models are optimized within the multitask framework. As discussed in the background section, an alternative approach to Autoregressive (AR) modeling is Non-Autoregressive (NAR) modeling. AR assumes that the output tokens are conditional dependent on the previously generated tokens. However, it causes significant latency during inference. NAR models solve this problem by outputting all the translated tokens in parallel thus speeding up the inference. Formally, they are given by (23)

   Report issue for preceding element

   $$p({\bf v}|{\bf u};\theta)=\prod_{t=1}^{T_{v}}p(v_{t}|,{\bf u};\theta)$$

   (23)

   There has been a surge in applying the non-autoregressive models (AR) in ASR and MT and it has prompted ST researchers to apply to it too. For example, (Inaguma et al., 2020a, 2021) trains NAR and autoregressive decoding conditioned on a shared speech encoder. Another line of NAR works (Chuang et al., 2021) explores CTC with ASR as an auxiliary task. CTC-based encoder only architecture ((Inaguma et al., 2020a, 2021) use encoder and decoder both) for NAR E2E ST task is shown to perform comparably or better than strong AR models in (Xu et al., 2023a).

   Report issue for preceding element

Discussion: Our study of Seq2Seq-based frameworks for ST task reveals that (a) structural bias can be obtained by stacked/pyramidal RNN and alignment smoothing, (b) regularizers such as transitivity and invertibility improves Character Error Rate, (c) HLR helps in transcription as well as translation, and (d) changing the self-attention of the encoder with a logarithmic distance penalty enhances translation, (e) Progressive training needs a huge data and training time to achieve superior results, and (f) multitask pre-training can be used to leverage unlabeled speech data. (Zhang et al., 2022a) shows that ST models trained from scratch using only ST tasks perform on par with or surpass pre-trained models. To achieve such results, proposed best practices include a smaller vocabulary, a wider feedforward layer, a deep speech encoder with the post-layer norm, CTC-based regularization, and parameter-distance penalty. Pre-training is still useful in low-resource data regimes. Transferring knowledge via KD from pre-trained MT to ST causes gender bias, omission of sentences, and generic verbal-tense choice. Use of large vocabulary and models is effective for NAR E2E ST task (Inaguma et al., 2020a). It indicates that leveraging NAR with LLMs may be a future direction to explore.

Streaming frameworks for ST tasks start outputting target tokens on seeing only partial inputs, that is, the translation of the input as soon as it arrives without waiting for the entire input. They are also known as Simultaneous ST (SimulST or SST)⁸⁸8Note that in MT literature, some works such as (Iranzo-S’anchez et al., 2022) differentiate between Streaming and Simultaneous setting where sentences are treated independently from each other. However, in ST, we find that existing works make no differentiation between them. (Goldman-Eisler, 1972; Fügen et al., 2007; Tsiartas et al., 2013; Grissom II et al., 2014). It finds application in online speech translation and video dubbing, to name a few. Traditionally, the streaming ST problem has been solved by feeding the segmented output of a streaming ASR model to a streaming MT model (Oda et al., 2014; Iranzo-Sánchez et al., 2020). However, due to the cascade nature of the model, it is prone to high latency and error propagation (Arivazhagan et al., 2019b, 2020; Zaidi et al., 2022). The SST problem faces several issues in practical implementation; reordering, acoustic ambiguity, and variable speech rate, and long inputs being prominent among them. Our literature survey reveals that most of the existing works focus on handling long streaming inputs and therefore, the discussion underneath revolves around that. Other issues mentioned above may also be considered for designing practical SST models.

Existing streaming frameworks intervene Seq2Seq framework at various places to design SST models. These are (a) encoder-level, (b) decoder-level, and (c) input/latent-level.

1. 1.

   Encoder-level: SOTA SST models use transformers as encoders. Due to the self-attention operation which looks at the entire utterance, it is unsuitable for streaming inputs. There exist some works that design encoders specialized for streaming inputs. For example, augmented memory transformer (Wu et al., 2020; Ma et al., 2020c) splits the utterance $U$ into smaller-segments $S=[s_{1},\ldots]$. Each segment $s_{n}$ consists of left context $I_{n}$, main context $c_{n}$, and right context $r_{n}$. Self-attention is calculated at the segment level only thereby reducing the time complexity. Augmented memory propagates the information from one segment to the other. Incremental transformer (Zhang et al., 2020) leverages a unidirectional encoder based on unidirectional-attention with future context masked for handling streaming inputs.

   Report issue for preceding element
2. 2.

   Decoder-level: Instead of modifying encoders, some works such as (Dalvi et al., 2018; Liu et al., 2020a; Nguyen et al., 2021; Guo et al., 2024) propose incremental decoding (see fig. 7). In this framework, input speech is divided into fixed-size chunks and decoded every time a new chunk arrives. To avoid distractions from constantly changing hypotheses, selected chunk-level predictions are committed to and no longer modified. The decoding of the next chunk is conditioned by the predictions committed. Instead of conditioning on all chunk-level predictions, a prefix function is chosen to select a partial hypothesis because early chunks contain limited information (Liu et al., 2020a). There exist several strategies for choosing the prefix function. For example, Hold-$n$ and LA-$n$ (Liu et al., 2020a), SP-$n$ (Nguyen et al., 2021) and Regularized Batched Inputs (R-BI) (Guo et al., 2024). Of these, Hold-$n$ either withholds or deletes the last $n$ tokens in each chunk, LA–$n$ involves displaying the agreeing prefixes of $n$ consecutive chunks. SP-$n$ stands for shared prefix of all best-ranked hypotheses. Contrary to these, RB-I applies various augmentations to input chunks to achieve regularization and SOTA results on the IWSLT SimulST task.

   Report issue for preceding element
3. 3.

   Input/latent-level: Since speech input is too fine-grained, deciding when to READ and WRITE is challenging. The existing works introduce pre-decision module which segments the input speech at fixed-chunks (fixed) or word-boundary (flexible). Similarly, READ/WRITE policy can be fixed or adaptive (Ma et al., 2020b). Most research in SST concentrates on either improving speech encoding or pre-decision while relying on fixed policies such as wait-$k$. In this section, we discuss fixed and adaptive pre-decisions/policies. These techniques are combined with Seq2Seq frameworks to devise streaming ST models.

   Report issue for preceding element

   

   Report issue for preceding element

   

   Report issue for preceding element

   Report issue for preceding element

   Wait-$k$ policy (Ma et al., 2018) (shown in fig. 9) learns the parameters $\theta$ of the model by optimizing the negative log-likelihood $-\sum_{(\mathbf{u,v})\in D}\log p(\mathbf{v}|\mathbf{u};k;\theta)$, where $k$ is the number of segments to look before starting translation (see Fig. 9). The probability $p(\cdot)$ is calculated as

   Report issue for preceding element

   $$p({\bf v}|{\bf u};k;\theta)=\prod_{t=1}^{T_{v}}p(v_{t}|v_{<t},u_{t+k};\theta)$$

   (24)

   wait-$k$ policy guarantees that the model can look at $t+k-1$ speech segments while predicting token $v_{t}$ (Ren et al., 2020). However, one limitation of the wait-$k$ policy is that it fails to do a beam search while decoding except for long-tail (Ma et al., 2018). To solve this problem, (Zeng et al., 2021) proposes a wait-$k$ stride-$N$ policy. It essentially is a wait-$k$ policy with the addition of $N$ READ and WRITE operations until the end of the sentence after reading the first $k$-segments. To determine the $k$-segments, (Chen et al., 2021b) leverages streaming ASR to guide the direct simultaneous ST decoding via beam search.

   Report issue for preceding element

   

   Report issue for preceding element

   As discussed above, determining when to write is crucial for efficient SST. Contrary to wait-$k$ policy which is a fixed-policy, Segmentation can be performed on the embedded speech using CTC (Ren et al., 2020), attention mechanism (Papi et al., 2022b), or incremental BEAM search (Yan et al., 2023). Essentially, these works adapt offline ST to SST showing spectacular performance on benchmark datasets. Note that the models proposed in (Papi et al., 2022b; Yan et al., 2023) train models in a cascade manner while the inference is E2E. Another issue with the fixed policy is that the model can not speed up or slow down appropriately with the input types. Other examples of fixed-policy are Wait-If* (Cho and Esipova, 2016b) and Monotonic Chunkwise Attention (MoChA) (Chiu and Raffel, 2018) that has been used in simultaneous MT and may be explored for SST.

   Report issue for preceding element

   The works mentioned above require that encoded speech be segmented so that the decoder can apply the wait-$k$ policy. The goal of segmentation is to identify the word, sub-word, or phone boundary which are usually not even (due to silences, longer syllables, etc.). That means the number of acoustic units varies with time in each segment. Monotonic-segmented Streaming ST (MoSST) (Dong et al., 2022) is based on learning when to translate, which has a monotonic segmentation module located between the acoustic encoder and the transformer. It has an Integrate-and-Fire (IF) neuron (Abbott, 1999), which fires above a threshold when the context is developed. If the context is not developed, the neuron receives signals and accumulates the acoustic vectors, thus mimicking adaptive policy for READ-WRITE operation. IF strategy has shown impressive performance in simultaneous ASR (Dong and Xu, 2019) and ST (Chang and yi Lee, 2022). It can be used for monotonic segmentation of the speech input along with adaptive decision strategy (Dong et al., 2022). Another adaptive policy-based technique is Monotonic Infinite Lookback Attention (MILk) (Arivazhagan et al., 2019b) used in simultaneous MT can be explored for SST. It essentially is a Monotonic Attention mechanism (Raffel et al., 2017) that extends to infinite encoder states, theoretically, in the past and trains the MT model along with the MILk. It achieves better quality-latency trade-offs than MoCHA thanks to its soft attention to all the encoder states and hard attention. Monotonic Multihead Attention (MMA) (Ma et al., 2019) that extends MILK to multiple heads has been used for SST by (Ma et al., 2020b). Its variants Efficient MMA (Ma et al., 2023) solve numerical stability and biased monotonic alignment issues present in MMA but have not been explored for SST tasks. Adaptive segmentation based on an adaptive policy that takes into account acoustic features and translation history (called meaningful units) is another effective mechanism for SST (Zhang et al., 2022b).

   Report issue for preceding element

   Both fixed and adaptive policy mechanisms employ segmentation modules that are outside the translation module. As such, it breaks the acoustic integrity and potentially may drop the translation performance. Therefore, efforts such as (Zhang and Feng, 2023) propose differentiable segmentation (DiSeg) learned jointly with the translation model using expectation training. DiSeg essentially predicts a Bernoulli random variable $\sigma(FFN(u_{i}))$, via a feed-forward network (FFN), to decide when to segment. After segmentation, they apply segmented attention which combines unidirectional and bidirectional attention into one while masking future speech frames. Expectation training first constrains the number of segments followed by learning segmentation from the translation model both at semantic and acoustic levels (Zhang and Feng, 2023).

   Report issue for preceding element

The discussion so far covered encoder, and decoder level changes, and fixed and adaptive policies used for segmentation, to develop SST models within the Seq2Seq frameworks. Another way to design SST models is by Transduction. It is the process of mapping a sequence to another sequence (Jurafsky and Martin, 2008). A transducer is a special type of Seq2Seq model that solves a few inherent problems. For example, online processing of the long inputs and monotonic sequence alignment is the biggest problem with Seq2Seq models (Graves, 2012), solved by transducers. Below we discuss a special type of transducer called RNN-T and its improvements.

RNN-T is a transducer that can learn alignment between two sequences in an online/streaming fashion (Graves, 2012) as shown in fig. 10 (a). Formally, it learns the conditional probability $p({\bf v|u})$ by marginalizing all possible alignment paths $A(\bf u,v)$ including blank symbol $\phi$ as

RNN-T differs from Seq2Seq in the sense that it divides the decoder into a predictor and a joiner. The predictor takes the previous time step output and yields the representation to be consumed by the joiner along with the hidden representation of the input from the encoder. Since the predictor does not look at the input, it can be pre-trained on the text-only data in a low-data scenario. There have been several SST models proposed based on variants of RNN-T which we discuss next.

One of the main issues with RNN-T is the strict monotonic alignment between the input and output sequences which makes them unsuitable for tasks requiring reordering such as MT, ST, etc. For example, Cross-Attention Augmented Transducer (CAAT shown in fig. 10(b)) optimizes translation and policy models in tandem (Liu et al., 2021a). It eliminates the RNN-T’s strict monotonic restriction for reordering in the translation. Using transformers as encoders to reduce the multi-step memory footprint causes a significant delay for CAAT. The use of regularization terms and substantial hyperparameter adjustment are some other limitations of CAAT. An extension of it in (Xue et al., 2022) leverages Transformer Transducer (TT) networks with attention pooling for streaming E2E ST tasks. Attention divides the input audio into chunks of specific sizes. At any time, processing any input frame $\bf u_{t}$ can only see frames within its chunk and a fixed number of left chunks. By sharing the encoder, they also propose a variant to handle E2E ST tasks in multilingual settings. The adaptive READ and WRITE policy choices between encoder output and ground truth contribute to its success. The same authors (Wang et al., 2023) propose to combine the benefits of language-specific and language-agnostic encoders within the TT framework. A shared encoder takes LIDs as gating values and computes weights for each language through the source LID scheduling scheme. The empirical results demonstrate superior performance and a smaller number of trainable parameters than bilingual ST. Adaptive (dynamic) policy for segmenting speech input has recently been explored in a Seq2Seq transduction setting by (Tan et al., 2024). It essentially applies a cross-attention mechanism to decide when to segment the input followed by dynamic compression via anchor representation. Thus, it saves memory and achieves a better latency-quality trade-off.

Besides Transducer and Seq2Seq models, re-translation is another approach adapted for SST task by (Niehues et al., 2016, 2018; Arivazhagan et al., 2019a, 2020) though in a cascade setting. In this approach, the translated output can be re-generated after a fixed amount of time and displayed later for better quality. Though it reduces latency by being greedy to display the partial translation, the output is highly unstable and causes flickring effect. This may give rise to a bad user experience. To mitigate instability, (Arivazhagan et al., 2020) propose a metric called erasure which takes into the length of the suffix deleted during re-translation. Dynamic masking of MT output in a cascade of streaming ASR and MT for improving stability has been explored in (Yao and Haddow, 2020). Another approach to reducing instability is luminance contrast and the Discrete Fourier Transform used in (Liu et al., 2023).

Evaluation of SST models: SST models in the literature have been evaluated using the quality and latency metrics presented in §3. Often showing a trade-off between quality and latency. Most of the existing works attempt to balance the quality and latency ignoring the visualization and cognitive load on the viewer when displayed on a screen. Towards this end, (Papi et al., 2021b) emphasizes considering visualization as a metric to be evaluated along with the latency and quality. However, little effort has been made in this direction by the SST community. Therefore, we wish to draw the researcher’s attention to also consider visualization as an evaluation metric for SST models. Towards this end, (Liu et al., 2023) propose tokenized alignment, word updates with semantic similarity, and smooth animation of live captions. They find that it leads to a reduction in fatigue, and distractions while increasing the viewer’s reading comfort.

Discussion: SST is a challenging problem and in that, E2E SST poses a further impediment. Our findings suggest that using adaptive policy significantly improves the latency-quality trade-off. Learned policy mechanisms have been an ongoing research and adapting them for true long-form SST may open new possibilities. Exploring differentiable segmentation for long sequences is still tapped and requires more investigation. Re-translation is found to be on par with or better than SOTA streaming models (Arivazhagan et al., 2020) under a very low revision rate. Such a finding alludes to considering re-translation in an E2E SST system design.

A low-resource language (LRL) is one where speech and/or text data are scarcely available – usually not enough to pre-train Seq2Seq models. As such, LRLs present challenges of their own such as overfitting and poor generalization. This section will discuss works where ST models are developed especially for low-resource languages. The proposed models under this category have generic architecture as shown in Fig.11(a) which is similar to Seq2Seq ST models. We find the approaches mainly use pre-training the encoder on high-resource ASR data and subsequent fine-tuning on ST data. Another approach that has emerged in recent years to tackle LRL issues is SSL. For example, (Bansal et al., 2019) empirically demonstrates 100% performance improvement on ST tasks. They find that if the ASR language differs from the source and target languages, then pre-training on ASR data enhances ST task performance. Though the BLEU score is improved, the absolute BLEU score is only 7.1. In (Wang et al., 2022), the unsupervised ST is implemented for low-resource settings using pseudo-labels from unsupervised cascade models. SSL with discrete-speech unit (DSU) has been used to fine-tune the ST model on limited ST data (Lam et al., 2024).

Code-mix language refers to speech where one primary language is used, but words or phrases from other (embedded) languages are also included. This phenomenon arises from a multitude of challenges, encompassing ambiguous vocabulary, fluctuating lexical representations, intermingling of languages at the word level, redundancy, and alterations in word sequencing. Therefore, it is non-trivial to handle code-mixing while building ST models.

We find that there exist only a few works on code-mix ST. In (Weller et al., 2022), the code-mix dataset is created with the existing publicly available corpora Fisher (Cieri et al., 2004) and Miami⁹⁹9https://github.com/apple/ml-code-switched-speech-translation. As shown in Fig. 11(b), code-mix ST models feed language ID in addition to speech input to the encoder of the Seq2Seq model (Weller et al., 2022). The Wav2Vec 2.0, an acoustic encoder, and mBART, a multilingual decoder, are used for both languages with an attention layer applied for the embedded language. The use of multilingual encoders and decoders is a common practice while building code-mix ST models (Yang et al., 2023). In particular, self-supervised multilingual pre-training with adapters may be explored further.

There is an abundance of unlabeled speech and text data. Since manual annotation and creating a parallel corpus is costly, the natural instinct is to exploit unlabeled data for training ST models. This section reviews works where researchers make use of the unlabeled speech data to advance the ST task performance.

For unsupervised ST tasks, it is common to leverage large-scale self-supervised and semi-supervised learning. For example, speech encoders such as Wav2vec 2.0 have been pre-trained in a self-supervised manner on Librilight data (Kahn et al., 2019) and used by (Li et al., 2020; Wang et al., 2021b) whereas the decoder is randomly initialized. The entire model is optimized on CoVoST 2 ST data, and the encoder is frozen. Thereby, self-training is executed to generate pseudo-labels for Libri-light data. The Wav2Vec 2.0 is a “student” model which is fine-tuned with ground truth CoVoST 2 data and pseudo labels. Finally, a language model (LM) is trained on CommonCrawl data and combined with the ST model to generate text via beam-search decoding. Following along, for training the E2E model, (Wang et al., 2021b) produces pseudo-labels by cascading ASR, text de-normalization, and MT in an Unsupervised manner. Wav2Vec 2.0 and mBART are optimized for domain adaption using in-domain data (Li et al., 2020). According to experimental results, the proposed method is effective for E2E models without pre-training. However, between supervised and unsupervised pre-trained models, performance gap is encountered, which may be investigated in future works.

The multilingual ST model aims to translate from/to multiple speech input/output languages. It can be one of many-to-one, one-to-many, or many-to-many. The ST models solve multilinguality issues using mainly three approaches: (a) language ID, (b) dual-decoder, and (c) pre-trained models.

1. 1.

   Language ID (LID) is the identification label that allows one to identify the target language and explicitly translate the speech simultaneously. The existing works handle multilinguality using LID either with encoder or decoder. In (Inaguma et al., 2019), the model uses LID in the decoder for one-to-many and many-to-many translation. They demonstrate impressive performance in translation from high-resource to low-resource languages without using any transcript data from LRL. However, using the LID embedding in the decoder (Gangi et al., 2019) is shown to underperform than using it in the encoder. The author shows that LID can be either concatenated or merged with the inputs and, when pre-trained with ASR data, can result in superior performance than the one-to-one system. The model, however, performs poorly when trained on many unrelated target languages. One-to-many and many-to-one multilingual ST systems of (Wang et al., 2020c, a) provide a good set of baselines for research purposes.

   Report issue for preceding element
2. 2.

   Dual-decoder model is the transformer with two decoders, one for each ASR and ST, and the dual-attention mechanism. In (Le et al., 2020), a dual-decoder model is proposed to optimize it for ASR and ST tasks jointly. The author hypothesizes that a dual-attention mechanism can benefit each task by transferring knowledge instantly or in wait-$k$ policy mechanism. Their model generalizes earlier models proposed for one-to-many and bilingual ST models.

   Report issue for preceding element
3. 3.

   Pre-trained Multilingual Models use a pre-trained encoder and decoder for acoustic modeling and language modeling, respectively. In (Li et al., 2020; Tran et al., 2020), the author shows that efficiently fine-tuning mBART, which is a pre-trained multilingual decoder (Liu et al., 2020c) can achieve SOTA results on CoVoST data on zero-shot cross-lingual and multilingual translation tasks. Along similar lines, (Le et al., 2021) shows that inserting adapters in between layers of the encoder-decoder framework and tuning them can improve the ST task performance over bilingual ST models. SeamlessM4T (Barrault et al., 2023), Whisper (Radford et al., 2023), and other foundation models are built using many of these concepts like language ID in the decoder, multilingual, multimodal, and multitask pre-training.

   Report issue for preceding element