End-to-End Speech-to-Text Translation: A Survey
端到端语音到文本翻译研究综述

Speech data can be augmented in various ways. For example, by adding noise, speed and pitch perturbation, time and frequency masking to name a few. SpeechAugment (Park et al., 2019) policy consists of warping the features, masking blocks of frequency channels, and time steps. It has been successfully used both for ASR (Vincent et al., 2017) and ST tasks (Bahar et al., 2019b). MixSpeech (Meng et al., 2021) as shown in Fig. 4(a) takes the weighted combination of two different speech features as input and two recognition losses with the same weights. A generalization of MixSpeech (Xie and Hansen, 2023) called MixRep applies the mixup idea to the acoustic feature and hidden layers inputs. MixRep combination with a regularization term along the time axis further improves ASR performance. Both MixSpeech and MixRep have been shown to perform well for low-resource ASR and their effectiveness is still to be tested for ST tasks. M3ST (Cheng et al., 2022) applies two levels of Fine-Tuning (FT) using mixup data– word, sentence, and frame level mix data in the first FT level and source speech and transcription mixup in the second FT level. M3ST achieves SOTA on MuST-C compared to baselines.
语音数据可通过多种方式进行增强，例如添加噪声、速度和音高扰动、时频掩蔽等。SpeechAugment（Park 等人，2019）策略包含特征扭曲、频率通道块掩蔽和时间步掩蔽，已成功应用于自动语音识别（Vincent 等人，2017）和语音翻译任务（Bahar 等人，2019b）。如图 4(a)所示，MixSpeech（Meng 等人，2021）采用两种不同语音特征的加权组合作为输入，并施加相同权重的双重识别损失。其泛化版本 MixRep（Xie 和 Hansen，2023）将混合增强思想应用于声学特征和隐藏层输入，结合时间轴正则化项进一步提升了语音识别性能。MixSpeech 与 MixRep 在低资源语音识别中表现优异，但其在语音翻译任务中的有效性仍有待验证。M3ST（Cheng 等人，2022）采用两级微调：第一级使用词汇/句子/帧级混合数据，第二级采用源语音与转写文本的混合增强。在 MuST-C 基准测试中，M3ST 取得了当前最优性能。

It is possible to augment both speech and text simultaneously and create new paired data. For example, sample, translate, and recombine (Lam et al., 2022b) first samples a suffix replacement from suffix memory corresponding to a pivot token from transcription. It then translates the combined new utterance (prefix+pivot+replacmenet suffix) to generate a new target sentence. The corresponding audio pair is obtained by concatenating the audio frames of the prefix, pivot, and replacement suffix. The interesting thing about the proposed method is that it can generate real-looking sentences contrary to pseudo-sentences. Concatenation of original ST data has been used to augment the entire training data (Lam et al., 2022a). In particular, (Lam et al., 2022a) proposes CatSpeaker that uses single speaker information and CatRandom that randomly generates audio-text pairs spoken by different speakers.
可以同时对语音和文本进行增强并创建新的配对数据。例如，采样-翻译-重组方法（Lam 等人，2022b）首先从后缀存储器中采样与转录文本中的枢轴标记相对应的后缀替换项，随后将组合后的新话语（前缀+枢轴+替换后缀）翻译生成新的目标语句。对应的音频对则是通过拼接前缀、枢轴及替换后缀的音频帧获得。该方法的有趣之处在于能生成真实感语句而非伪句子。原始语音翻译数据的串联拼接已被用于增强整个训练集（Lam 等人，2022a）。具体而言，（Lam 等人，2022a）提出了使用单一说话者信息的 CatSpeaker 方案，以及随机生成不同说话者音频-文本对的 CatRandom 方案。

ST models often use ASR transcripts and MT translations as auxiliary data which needs to be fed to the encoder and decoder, respectively. To learn representation for such text data, existing works rely on word embedding (Zhang et al., 2023c; Bérard et al., 2016), LSTM (Kim et al., 2017; Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019), and Transformer (Wang et al., 2021b; Liu et al., 2021a; Zeng et al., 2021). Text data is often tokenized and fed as either a word or as a character (Bérard et al., 2018). The output of the decoder could be graphene, characters, or words.
语音转文本模型常将自动语音识别(ASR)转录文本和机器翻译(MT)译文作为辅助数据，分别输入编码器和解码器。为学习此类文本数据的表征，现有研究主要采用词嵌入(Zhang et al., 2023c; Bérard et al., 2016)、长短期记忆网络(Kim et al., 2017; Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019)以及 Transformer 架构(Wang et al., 2021b; Liu et al., 2021a; Zeng et al., 2021)。文本数据通常经过分词处理后，以单词或字符形式输入(Bérard et al., 2018)。解码器输出可以是字形、字符或单词。

ST models take speech as input and utilize various speech-based feature representation methods to convert speech into a vector representation. Traditional speech feature extraction methods such as Perceptual Linear Prediction, (PLP), Fbank, and Mel-Filter Cepstral Coefficient (MFCC) (Rabiner and Schafer, 2010) have been used after normalization to extract speech features by many (Duong et al., 2016; Bérard et al., 2016; Kim et al., 2017; Bérard et al., 2018; Anastasopoulos and Chiang, 2018; Bansal et al., 2019; Jia et al., 2019; Inaguma et al., 2019; Liu et al., 2020d; Dong et al., 2021; Le et al., 2023b; Parcollet et al., 2024), sometimes combining them with pitch features and speech augmentation methods as described in §5. These feature extraction methods are sometimes being replaced by distributed feature representation methods such as speech word2vec (Chung and Glass, 2018) owing to their dense continuous feature representation capability.
语音到文本（ST）模型以语音作为输入，采用多种基于语音的特征表示方法将语音转换为向量表征。传统语音特征提取方法如感知线性预测（PLP）、滤波器组（Fbank）和梅尔频率倒谱系数（MFCC）（Rabiner and Schafer, 2010）经过归一化处理后，已被众多研究者（Duong et al., 2016; Bérard et al., 2016; Kim et al., 2017; Bérard et al., 2018; Anastasopoulos and Chiang, 2018; Bansal et al., 2019; Jia et al., 2019; Inaguma et al., 2019; Liu et al., 2020d; Dong et al., 2021; Le et al., 2023b; Parcollet et al., 2024）用于语音特征提取，有时会如第 5 节所述结合基频特征与语音增强方法使用。由于具备稠密连续特征表示能力，这些特征提取方法正逐渐被语音 word2vec（Chung and Glass, 2018）等分布式特征表示方法所替代。

It is difficult to get a large amount of labeled speech data to learn supervised speech feature representation. Therefore, more recent works exploit speech features learned via unsupervised and self-supervised ways, mapping continuous speech signal to discrete units– akin to words and sub-words in the text domain. Such a representation facilitates tools developed in NLP to borrow in the speech domain. Among them the most popular is Wav2Vec (Schneider et al., 2019) and its variants such as w2v-BERT (Chung et al., 2021) and Wav2vec 2.0 (Baevski et al., 2020) used in (Tran et al., 2020; Le et al., 2020; Li et al., 2020; Han et al., 2021; Popuri et al., 2022; Zhang et al., 2023c). Interestingly, Wav2Vec and its variants can be used as an encoder in a Seq2Seq framework alone or combined with adapters and CNN for Length shrinking³³3Length shrinking is an important issue in ST task since speech is a much longer sequence than text. Therefore, existing works employ various techniques such as length adapters, CNN, CTC for length shrinking. A few works such as CSTNet (Khurana et al., 2020; Wang et al., 2020d) use CNN for feature extraction and length shrinking.
获取大量标注语音数据以学习有监督的语音特征表示存在困难。因此，近期研究多采用无监督和自监督方式学习语音特征，将连续语音信号映射为离散单元——类似于文本领域的单词和子词。这种表征方式便于将自然语言处理领域的工具迁移至语音领域。其中最流行的是 Wav2Vec（Schneider 等人，2019）及其变体，如 w2v-BERT（Chung 等人，2021）和 Wav2vec 2.0（Baevski 等人，2020），这些模型已被应用于（Tran 等人，2020；Le 等人，2020；Li 等人，2020；Han 等人，2021；Popuri 等人，2022；Zhang 等人，2023c）等研究。值得注意的是，Wav2Vec 及其变体可单独作为 Seq2Seq 框架中的编码器，也可与适配器和 CNN 结合用于长度缩减 ³ 。部分研究如 CSTNet（Khurana 等人，2020；Wang 等人，2020d）则采用 CNN 进行特征提取和长度缩减。

More recent works in ST are employing HuBERT (Hsu et al., 2021) for speech representation (among other benefits of HuBERT) (Zhang et al., 2023a). Hubert offers stable training and better targets than Wav2Vec 2.0 since it uses hidden layers representation during the clustering process. For encoding long-speech signals, Conformers (Gulati et al., 2020) can be used as they provide local context through convolution block and global context through an attention mechanism. Seamless4MT (Barrault et al., 2023) uses conformer for speech encoding.
近期语音翻译研究开始采用 HuBERT（Hsu 等人，2021）作为语音表征方法（Zhang 等人，2023a），这得益于 HuBERT 的多重优势。相较于 Wav2Vec 2.0，HuBERT 在聚类过程中采用隐藏层表征，能提供更稳定的训练目标和更优的性能指标。针对长语音信号编码，可选用 Conformer 模型（Gulati 等人，2020），其通过卷积模块捕获局部上下文，并借助注意力机制获取全局上下文。Seamless4MT 系统（Barrault 等人，2023）即采用 Conformer 架构进行语音编码。

Other speech representation techniques such as VQ-VAE (van den Oord et al., 2017), WavLM (Chen et al., 2022), data2vec (Baevski et al., 2022), Robust data2vec (Zhu et al., 2023), SpeechLM (Zhang et al., 2024b), may also be explored while encoding speech for ST tasks.
其他语音表征技术，如 VQ-VAE（van den Oord 等，2017）、WavLM（Chen 等，2022）、data2vec（Baevski 等，2022）、Robust data2vec（Zhu 等，2023）、SpeechLM（Zhang 等，2024b）等，在语音到文本翻译任务的编码过程中同样值得探索。

The speech and text in an ST task are semantically related because both of them refer to the same thing. Therefore, it is imperative to learn a joint speech-text representation in the hope of bridging the modality gap between them. A method for learning a combined representation of text and speech is called modality bridging (see fig.5). Hence, a good ST model should learn a representation such that embeddings of both modalities for similar speech-text pairs lie close to each other. It is believed that low performance on ST tasks is due to models not learning aligned representations of speech and text. Therefore, different authors have devised different ways to fill the gap, which fall into five major approaches: (a) adapters, (b) contrastive learning, (c) knowledge-distillation, (d) optimal transport, and (e) mix-up strategy. Below we discuss the works utilizing these approaches and show the pros and cons.
在语音到文本（ST）任务中，语音与文本具有语义关联性，因为二者指向同一事物。因此，学习联合语音-文本表征以弥合模态差异至关重要。这种学习语音与文本联合表征的方法称为模态桥接（见图 5）。理想的 ST 模型应习得一种表征方式，使得相似语音-文本对的两种模态嵌入在空间中彼此接近。当前 ST 任务性能不佳的主要原因被认为是模型未能学习到语音与文本的对齐表征。为此，不同研究者提出了填补这一差距的五类主要方法：(a)适配器、(b)对比学习、(c)知识蒸馏、(d)最优传输、(e)混合策略。下文将分别探讨采用这些方法的研究成果，并分析其优劣。

1. 1.

   Adapters are small modules integrated with pre-trained networks for specific tasks (Houlsby et al., 2019). They have performed on par with fine-turning-based approaches while requiring only a fraction of trainable parameters. For example, in (Gállego et al., 2021; Zhao et al., 2022; Sarkar et al., 2023), the modality gap is filled using adapter layers, which is a multi-headed self-attention with pooling operation. The author uses Wav2Vec 2.0 (Baevski et al., 2020) for speech-feature extraction, wherein self-attention layers in the transformer are equipped with pooling operation for dimensionality reduction to match the text representation.

   Report issue for preceding element
   
   1. 适配器是与预训练网络集成的小型模块，用于特定任务（Houlsby 等人，2019）。其性能与基于微调的方法相当，但仅需少量可训练参数。例如在（Gállego 等人，2021；Zhao 等人，2022；Sarkar 等人，2023）中，研究者使用带池化操作的多头自注意力适配器层来填补模态鸿沟。作者采用 Wav2Vec 2.0（Baevski 等人，2020）进行语音特征提取，其中变压器的自注意力层配备池化操作以实现降维，从而匹配文本表征。
2. 2.

   Contrastive learning approximates the “semantic” distance in the input space using a simple distance in the target space after mapping input patterns onto the target space (Chopra et al., 2005). It tries to bring positive instances closer while pushing negative ones apart. It has been used excessively in both supervised and unsupervised settings for learning representations. For example, (Zhang et al., 2023c) performs the explicit knowledge transfer through contrastive learning. It learns frame and sentence-level speech feature representation and uses whitening (Su et al., 2021) to alleviate the MT representation degeneration. (Liu et al., 2019) decouples the encoder representation into three parts: acoustic encoder, shrinking (done via CTC) of acoustic encoder output, and semantic encoder for modality-gap bridging. Using a contrastive learning architecture, Chimera (Han et al., 2021) trains a semantic memory module which is shared for overcoming the modality distance. XSTNet (Ye et al., 2021) augmented with contrastive loss (Ye et al., 2022a) investigates three different methods: span masked representation, word-repetition and cut-off. It claims that contrastive loss is better than CTC and L2 loss. Word-aligned contrastive learning (WACO) (Ouyang et al., 2023) bridges the modality gap by forming average speech and word embedding of the same word as the positive pair while of different words as negative pairs. CSTNet is a self-supervised learning framework based on contrastive learning (using a mix of triplet losses)(Khurana et al., 2020). On top of the CTC loss, the boundary-based speech length shrinking mechanism is applied in (Zeng et al., 2022). The authors claim that if boundary-based shrinking is applied with other modality-bridging techniques, such as contrastive loss, it can further improve the model performance. The approach presented achieves lower inference speed and memory footprint. (Yin et al., 2023) proposes a novel integration of speech and text, referred to as a third modality. This fusion is achieved through the application of Cross-modal Contrastive Learning (Sohn, 2016) and Cross-Attentive Regularization (Tang et al., 2021a). Additionally, the method incorporates techniques such as Knowledge Distillation and Jensen-Shannon Divergence (Lin, 1991; Liu et al., 2019; Gaido et al., 2020a) to bridge the modality gap, addressing challenges related to input representation, semantics, and hidden states.
   对比学习通过将输入模式映射到目标空间后，利用目标空间中的简单距离来近似输入空间的"语义"距离（Chopra 等，2005）。该方法致力于拉近正样本间距同时推远负样本，已在监督与非监督场景中被广泛用于表征学习。例如（Zhang 等，2023c）通过对比学习实现显式知识迁移，其同时学习帧级和句子级语音特征表示，并采用白化处理（Su 等，2021）缓解机器翻译表征退化问题。（Liu 等，2019）将编码器表征解耦为三部分：声学编码器、声学编码输出的 CTC 收缩模块以及用于模态间隙桥接的语义编码器。Chimera（Han 等，2021）采用对比学习架构训练共享语义记忆模块以克服模态距离。增强对比损失的 XSTNet（Ye 等，2021；Ye 等，2022a）探索了三种方法：跨度掩码表示、词语重复与截断策略。 研究表明对比损失函数优于 CTC 和 L2 损失函数。词对齐对比学习（WACO）（Ouyang 等，2023）通过将同一词语的语音嵌入向量与文本嵌入向量的均值构成正样本对，不同词语的嵌入向量构成负样本对，从而弥合模态差异。CSTNet 是基于对比学习的自监督框架（采用混合三元组损失函数）（Khurana 等，2020）。在 CTC 损失函数基础上，（Zeng 等，2022）应用了基于边界的语音长度压缩机制。作者指出，若将基于边界的压缩机制与其他模态桥接技术（如对比损失）结合使用，可进一步提升模型性能。该方法实现了更低的推理时延和内存占用。（Yin 等，2023）提出了一种称为"第三模态"的语音文本融合新范式，通过跨模态对比学习（Sohn，2016）和交叉注意力正则化（Tang 等，2021a）实现模态融合。 此外，该方法融合了知识蒸馏与 Jensen-Shannon 散度（Lin, 1991；Liu 等, 2019；Gaido 等, 2020a）等技术以弥合模态差异，解决了输入表征、语义及隐状态相关的挑战。

   Report issue for preceding element

   Models/Techniques  模型/技术	Problem Solved  问题解决	Dataset	Language Pair  语言对	Speech (hours)	Metric (BLEU)
   M-Adapter + W2V2 + mBart (Baevski et al., 2020)	training gap between Pre-training & Fine-tuning the modality	MuST-C	En$\rightarrow$De	408	25.9
   			En$\rightarrow$Ro	432	24.62
   			En$\rightarrow$Fr	492	37.34
   Chimera (Han et al., 2021)	projecting audio & text to a common semantic representation	MuST-C	En$\rightarrow$De	408	27.1
   			En$\rightarrow$Fr	492	35.6
   			En$\rightarrow$Ru	489	17.4
   			En$\rightarrow$Es	504	30.6
   			En$\rightarrow$It	465	25.0
   			En$\rightarrow$Ro	432	24.0
   			En$\rightarrow$Pt	385	30.2
   			En$\rightarrow$Nl	442	29.2
   ConST (XSTNet + Constrastive Loss) (Ye et al., 2021)	closes modality gap	MuST-C	En$\rightarrow$De	408	28.3
   			En$\rightarrow$Es	504	32.0
   			En$\rightarrow$Fr	492	38.3
   			En$\rightarrow$It	465	27.2
   			En$\rightarrow$Nl	442	31.7
   			En$\rightarrow$Pt	385	33.1
   			En$\rightarrow$Ro	432	25.6
   			En$\rightarrow$Ru	489	18.9
   W2V2 + mBart + Adapter (Gállego et al., 2021; Zhao et al., 2022)	slow convergence speed	MuST-C	En$\rightarrow$De	408	28.22
   WACO (Ouyang et al., 2023)	limited parallel data (1-hour)	MuST-C	En$\rightarrow$De	1	17.5
   AdaTrans (Zeng et al., 2022)	closing gap between length of speech & text	MuST-C	En$\rightarrow$De	408	28.7
   			En$\rightarrow$Fr	492	38.7
   			En$\rightarrow$Ru	489	19.0
   STEMM (Fang et al., 2022)	Speech representation	MuST-C	En$\rightarrow$De	408	28.7
   			En$\rightarrow$Fr	492	37.4
   			En$\rightarrow$Ru	489	17.8
   			En$\rightarrow$Es	504	31.0
   			En$\rightarrow$It	465	25.8
   			En$\rightarrow$Ro	432	24.5
   			En$\rightarrow$Pt	385	31.7
   			En$\rightarrow$Nl	442	30.5
   CTC loss + Optimal Transport (Siamese-PT) (Le et al., 2023b)	without change in architecture	MuST-C	En$\rightarrow$De	408	27.9
   			En$\rightarrow$Es	504	31.8
   			En$\rightarrow$Fr	492	39.2
   			En$\rightarrow$It	465	27.7
   			En$\rightarrow$Nl	442	31.7
   			En$\rightarrow$Pt	385	34.2
   			En$\rightarrow$Ro	432	27.0
   			En$\rightarrow$Ru	489	18.5
   Fine & Coarse Granularity Contrastive Learning (Zhang et al., 2023c)	limited knowledge transfer ability	MuST-C	En$\rightarrow$De	408	29.0
   			En$\rightarrow$Fr	492	38.3
   			En$\rightarrow$Ru	489	19.7
   			En$\rightarrow$Es	504	31.9
   			En$\rightarrow$It	465	27.3
   			En$\rightarrow$Ro	432	26.8
   			En$\rightarrow$Pt	385	32.7
   			En$\rightarrow$Nl	442	31.6

   Report issue for preceding element
3. 3.

   Knowledge-distillation (Hinton et al., 2015) is a mechanism to distill information from a trained and large “teacher” model to a smaller and efficient “student” model. It has been used with $L_{2}$ loss in (Huzaifah and Kukanov, 2023) to address the modality gap issue.

   Report issue for preceding element
   
   3. 知识蒸馏（Hinton 等人，2015）是一种将训练好的大型"教师"模型中的信息提炼到更小更高效的"学生"模型中的机制。(Huzaifah 和 Kukanov，2023)中采用 $L_{2}$ 损失函数来应用该机制，以解决模态差异问题。
4. 4.

   Optimal transport (OT) (Peyré et al., 2019) is a mechanism for comparing two probability distributions . In the ST task, speech and text representations may be deemed as two probability distributions, and therefore, OT can be applied. More formally, suppose $\alpha$ and $\beta$ denote the discrete probability distributions corresponding to speech and text representations. The masses at each position $u_{i}$ and $v_{i}$ are $a_{i}$ and $b_{j}$ respectively such that $\sum_{i=1}^{m}a_{i}=1$ and $\sum_{j=1}^{n}b_{j}$. Suppose further that the cost of transporting a unit of mass from $u_{i}$ to $v_{j}$ is $c(u_{i},v_{j})$, where $c$ is some cost function such as Euclidean distance. Let $Z_{ij}\geq 0$ be the quantity of mass to be transported from $u_{i}$ to $v_{j}$ then the goal of OT is to move all masses from $\alpha$ to $\beta$ such that the following objective function is minimized

   Report issue for preceding element
   
   4. 最优传输（OT）（Peyré等人，2019）是一种比较两个概率分布的机制。在语音翻译任务中，语音和文本表征可被视为两种概率分布，因此可应用 OT 方法。更形式化地说，假设 $\alpha$ 和 $\beta$ 分别表示语音和文本表征对应的离散概率分布，其中各位置 $u_{i}$ 和 $v_{i}$ 的质量分别为 $a_{i}$ 和 $b_{j}$ ，且满足 $\sum_{i=1}^{m}a_{i}=1$ 和 $\sum_{j=1}^{n}b_{j}$ 。进一步假设将单位质量从 $u_{i}$ 传输到 $v_{j}$ 的成本为 $c(u_{i},v_{j})$ ，其中 $c$ 为某种成本函数（如欧氏距离）。设 $Z_{ij}\geq 0$ 为从 $u_{i}$ 传输到 $v_{j}$ 的质量，则 OT 的目标是通过将所有质量从 $\alpha$ 转移到 $\beta$ ，使得以下目标函数最小化。

   $$\min_{Z}\langle C,Z\rangle,\qquad Z{\bf 1}_{n}=a,Z^{T}{\bf 1}_{m}=b,Z\geq 0$$

   (21)

   In the above eq., $C$ and $Z$ are the matrices whose elements are $c_{ij}=c(u_{i},v_{j})$ and $Z_{ij}$, respectively. ${\bf 1}$ denotes the vector of ones. In ST task, $c(u_{i},v_{j})=\|u_{i}-v_{j}\|_{p}$ for some $p\geq 1$. The loss corresponding to (21) is called Wassertein loss optimizing which is costly. Hence an entropy-regularized upper-bound approximation is often optimized
   在上述方程中， $C$ 和 $Z$ 分别是元素为 $c_{ij}=c(u_{i},v_{j})$ 和 $Z_{ij}$ 的矩阵。 ${\bf 1}$ 表示全 1 向量。在语音翻译任务中，对于某些 $p\geq 1$ 有 $c(u_{i},v_{j})=\|u_{i}-v_{j}\|_{p}$ 成立。与式(21)对应的损失函数称为 Wasserstein 损失，其优化过程计算代价高昂。因此通常采用熵正则化上界近似进行优化。

   Report issue for preceding element

   $$\min_{Z}\{\langle C,Z\rangle-\lambda H(Z)\}$$

   (22)

   where $\lambda$ is a regularization parameter and $H(\cdot)$ is the von-Neuman entropy matrix.
   其中 $\lambda$ 是正则化参数， $H(\cdot)$ 为冯·诺依曼熵矩阵。

   Report issue for preceding element

   Recent works make use of the OT as presented above. For example, (Le et al., 2023b) uses optimal transport and CTC together to close the modality gap during the pre-training phase. They show significant gains in BLEU score when the ST model is fine-tuned without any external data compared to multitask learning. Similarly, (Tsiamas et al., 2024, 2023) uses OT+CTC to align the speech-encoder representation space with the MT embedding space whereas (Zhou et al., 2023) aligns the two representations via OT followed by cross-modal mix-up at the token level.
   近期研究采用了上述最优传输方法。例如，(Le 等人，2023b) 将最优传输与 CTC 结合使用，在预训练阶段缩小模态差异。实验表明，与多任务学习相比，未经外部数据微调的语音翻译模型 BLEU 分数显著提升。类似地，(Tsiamas 等人，2024，2023) 采用 OT+CTC 方法对齐语音编码器表征空间与机器翻译嵌入空间，而(Zhou 等人，2023) 则通过最优传输实现表征对齐后，在 token 级别进行跨模态混合增强。

   Report issue for preceding element
5. 5.

   Mix-up strategy: Speech-Text Manifold Mixup (STEMM) (Fang et al., 2022) strategy uses speech embedding. It mixes embeddings of speech and text into the encoder-decoder of a translation model for bridging the modality gap under the self-supervised learning framework. PromptST (Yu et al., 2023) presents a linguistic probing learning strategy, referred to as Speech-Senteval, inspired by the approach introduced by (Conneau et al., 2018). This strategy is implemented on the higher layer of the encoder within pre-trained ST models, specifically targeting the challenges associated with learning linguistic properties that these models often struggle with at the higher layers.

   Report issue for preceding element
   
   5. 混合策略：语音-文本流形混合（STEMM）（Fang 等人，2022）策略采用语音嵌入技术。该策略将语音和文本的嵌入混合到翻译模型的编码器-解码器中，以在自监督学习框架下弥合模态差异。PromptST（Yu 等人，2023）提出了一种称为 Speech-Senteval 的语言探测学习策略，其灵感源自（Conneau 等人，2018）提出的方法。该策略在预训练语音翻译模型编码器的高层实施，专门针对这些模型在高层学习语言属性时常见的困难。

   Table 1 presents the performance scores of ST models based on modality-bridging techniques. We can observe that mixup strategy achieves the highest BLEU score on En-De pair. Whereas boundary-based speech length shrinking mechanism matches the score when combined with other modality-bridging techniques.
   表 1 展示了基于模态桥接技术的语音翻译模型性能评分。我们可以观察到，混合策略在英德语言对上获得了最高的 BLEU 分数。而当边界语音长度压缩机制与其他模态桥接技术结合使用时，其得分与之相当。

   Report issue for preceding element

Discussion: The study finds that adapters can shrink the speech length as well as the modality distance between the text and speech representations while requiring a small number of trainable parameters. The contrastive loss is found to be better than CTC and $L_{2}$ loss for modality-bridging. The boundary-based speech length shrinking combined with contrastive loss may improve the ST task performance. Finally, it is possible to build ST models requiring zero parallel ST data (Tsiamas et al., 2024).
讨论：研究发现，适配器不仅能缩减语音长度，还能缩小文本与语音表征之间的模态距离，同时仅需少量可训练参数。对比损失函数在模态桥接方面表现优于 CTC 和 $L_{2}$ 损失。基于边界的语音长度缩减与对比损失相结合，可提升语音翻译任务的性能。最终研究表明，构建无需平行语音翻译数据的语音翻译模型具有可行性（Tsiamas 等人，2024 年）。

Offline frameworks perform ST tasks where output tokens are produced after having seen the entire speech utterance. These frameworks heavily rely on Seq2Seq architecture as shown in Fig. 6. It has an encoder for speech input, a decoder for text output, and an optional shared/semantic decoder connecting the encoder and the decoder. The model is usually optimized for the ST loss or sometimes in a multitask learning framework where ASR/MT/CTC (Graves et al., 2006) losses are combined with ST loss. Other times Transfer learning is utilized for leveraging pre-trained models for ST tasks. Another approach that has been gaining a lot of attention is Non-Autoregressive modeling (NAR) for the E2E ST task which gives faster inference. The following section will delve deeper into these approaches.
离线框架执行语音翻译任务时，需在接收完整段语音输入后才生成输出标记。如图 6 所示，这类框架主要基于序列到序列架构，包含语音输入的编码器、文本输出的解码器，以及连接二者的可选共享/语义解码器。模型通常针对语音翻译损失函数进行优化，有时也会采用多任务学习框架，将自动语音识别/机器翻译/连接时序分类（Graves 等人，2006）等损失函数与语音翻译损失结合。此外，迁移学习也常被用于借助预训练模型完成语音翻译任务。近年来，端到端语音翻译中非自回归建模方法因能实现更快推理而备受关注，下文将对这些方法进行深入探讨。

The Seq2Seq-based ST models proposed in the literature either use specialized encoders such as transformers or attention mechanisms which we discuss next.
文献中提出的基于 Seq2Seq 的语音翻译模型主要采用两种架构：一种是专用编码器（如 Transformer 结构），另一种是注意力机制，我们将在下文详细探讨。

1. 1.

   Attention mechanism is used to concentrate on specific sections of the input data instead of the entire data (Larochelle and Hinton, 2010; Mnih et al., 2014; Vaswani et al., 2017). It has been a successful strategy for getting state-of-the-art (SOTA) results in NLP, computer vision, and other areas. There exist various types of attention in the literature such as soft, hard, local, monotonic, multihead, self- and cross-attention, inter alia. For more details, interested readers are encouraged to skim through (Mnih et al., 2014; Vaswani et al., 2017; Brauwers and Frasincar, 2022). Below we provide efforts made to handle ST tasks using the attention mechanism within the Seq2Seq framework.

   Report issue for preceding element
   
   1. 注意力机制用于聚焦输入数据的特定部分而非整体数据（Larochelle & Hinton, 2010; Mnih et al., 2014; Vaswani et al., 2017）。该机制已成为自然语言处理、计算机视觉等领域取得最先进（SOTA）成果的成功策略。现有文献中记载的注意力类型包括但不限于：软注意力、硬注意力、局部注意力、单调注意力、多头注意力、自注意力与交叉注意力等。更多细节可参阅（Mnih et al., 2014; Vaswani et al., 2017; Brauwers & Frasincar, 2022）。下文将重点介绍在序列到序列框架中运用注意力机制处理语音翻译任务的研究进展。

   The convolutional attention to “remember” and avoid translating the signal twice is used within Seq2Seq by (Bérard et al., 2016), which outperforms a hierarchical encoder with better results on synthetic data without using transcripts. The same author in (Bérard et al., 2018) uses source transcript and achieves results close to cascade models on LibriSpeech data. In (Duong et al., 2016), the author proposes phone-to-text alignment with a structural bias feature in the attention model. The measurement of alignment has been explored in (Anastasopoulos et al., 2016), which uses IBM’s translation model as well as dynamic time warping⁴⁴4dynamic time warping (DTW) is an algorithm for measuring similarity between two temporal sequences, which may vary in speed.. Seq2seq with attention trained using multitask learning achieves promising results in (Weiss et al., 2017). These models, however, struggle with noisy inputs and long acoustic signals (Kim et al., 2017). They use a joint CTC-attention model (Graves et al., 2006) trained through multitask learning by incorporating regularizers. The author uses two decoders where the second decoder seeks higher level representation (HLR) from the first decoder besides the encoder via the attention mechanism. Attention-Passing Model (APM) (Sperber et al., 2019), which only passes high-attention vectors from the audio encoder to the translation text for decoding demands a smaller amount of data for training.
   (Bérard 等，2016)在 Seq2Seq 模型中采用了卷积注意力机制来"记忆"并避免对信号进行二次翻译，该方法在不使用转录文本的情况下，于合成数据上取得了优于分层编码器的效果。同一作者在(Bérard 等，2018)中引入源语言转录文本，在 LibriSpeech 数据集上获得了接近级联模型的表现。(Duong 等，2016)提出在注意力模型中引入音素-文本对齐的结构偏置特征。(Anastasopoulos 等，2016)探索了对齐度量方法，同时采用 IBM 翻译模型和动态时间规整技术 ⁴ 。(Weiss 等，2017)通过多任务学习训练的注意力 Seq2seq 模型取得了显著成果。然而这些模型在处理噪声输入和长语音信号时仍存在困难(Kim 等，2017)。研究者采用(Graves 等，2006)提出的联合 CTC-注意力模型，通过引入正则化项进行多任务学习训练。该模型采用双解码器架构，其中第二解码器除通过注意力机制获取编码器信息外，还从第一解码器提取高层表征(HLR)。 注意力传递模型（APM）（Sperber 等人，2019 年）仅将音频编码器中的高注意力向量传递至翻译文本以供解码，该模型训练所需数据量较少。

   Report issue for preceding element
2. 2.

   Transformer is the architecture based on multi-headed self-attention (Vaswani et al., 2017) which produces contextualized representation of the input. Because of parallelization and contextual representation, transformers have outperformed RNNs on several NLP tasks. This entails us applying transformers for the ST task as well. Transformer-based Seq2Seq with attention is proposed in (Cattoni et al., 2021). The architecture has a quadratic memory complexity, which involves: (a) CNN to downsample the input, and (b) 2-D attention to address short-range dependencies of spectrograms. In (Alastruey et al., 2022), the weight of some attention is avoided for speech tasks, hence decreasing the size of the attention matrix. The transformer encodes the speech features, thereby introducing local self-attention with a suitable window size for each layer to reduce the computational complexity. Other transformer variants which reduce its quadratic complexity such as perceivers (Jaegle et al., 2021) have been used as an encoder (Tsiamas et al., 2022b). Besides quadratic complexity, transformers require lossy downsampling of speech features thus potentially throwing useful linguistic information. To tackle such issues, Speechformers have been proposed (Papi et al., 2021a) which aggregates information at higher layers based on more informed linguistic criteria.

   Report issue for preceding element
   
   2. Transformer 是一种基于多头自注意力机制的架构（Vaswani 等人，2017），可生成输入的上下文表征。得益于并行化处理和上下文表征能力，Transformer 在多项自然语言处理任务中已超越循环神经网络。这促使我们也将 Transformer 应用于语音翻译任务。基于 Transformer 的注意力序列到序列模型由（Cattoni 等人，2021）提出，该架构具有二次方内存复杂度，其特点包括：（a）使用卷积神经网络对输入进行下采样；（b）采用二维注意力机制处理声谱图的短程依赖关系。（Alastruey 等人，2022）通过避免部分注意力权重来缩减注意力矩阵规模，该 Transformer 通过编码语音特征，在各层引入具有合适窗口大小的局部自注意力以降低计算复杂度。其他降低二次方复杂度的 Transformer 变体，如感知器（Jaegle 等人，2021），已被用作编码器（Tsiamas 等人，2022b）。 除了二次复杂度问题外，变换器模型需要对语音特征进行有损下采样，这可能导致有用语言信息的丢失。为解决这一问题，研究者提出了 Speechformer 模型（Papi 等人，2021a），该模型基于更精确的语言学标准在高层网络进行信息聚合。

As discussed earlier, multitask learning combines the optimization of ST loss with an auxiliary loss such as ASR/MT/CTC loss. Another direction that has been explored by ST researchers is transfer learning in that Seq2Seq encoder/decoders are first pre-trained using ASR/MT data respectively and then the entire model is fine-tuned using ST data. Below, we discuss works based on multitask/transfer learning frameworks.
如前所述，多任务学习将语音翻译损失与辅助损失（如自动语音识别/机器翻译/连接时序分类损失）联合优化。语音翻译研究探索的另一方向是迁移学习——先分别使用语音识别/机器翻译数据对序列到序列的编码器/解码器进行预训练，再用语音翻译数据对整个模型进行微调。下文将讨论基于多任务/迁移学习框架的研究成果。

1. 1.

   ST with ASR: ST with ASR models make use of the transcript data along with speech-text pairs for pre-training. For example, curriculum pre-training (Wang et al., 2020d) refers to using ASR data for pre-training a Seq2Seq model, allowing it to learn transcription. The author argues that if the model is further pre-trained on learning semantic concepts (via frame-based masked language modeling) and word alignment (via frame-based bilingual lexical translation), it boosts the ST task performance. Specifically, existing E2E models either pre-train the encoder or use multi-task learning for ST tasks. As such, the encoder cannot isolate the learning of three tasks: transcription, semantic concept, and alignment, which are segregated by dividing the labor, and experiments prove the theoretical claims. Listen, Understand, and Translate (LUT) (Dong et al., 2021) uses the Seq2Seq model with external ASR loss. Their primary contribution is to introduce a semantic encoder network, whose task is to use the encoder’s output from transcription to minimize the mean-squared loss between the semantic representations and the BERT embeddings of the target text. Such a strategy implicitly builds and trains an NMT model for translation. Pre-training using ASR and/or MT has also been found useful in low-resource scenarios (Zhang et al., 2022a).

   Report issue for preceding element
   
   1. 基于 ASR 的语音翻译：此类模型利用转录文本数据及语音-文本对进行预训练。例如课程式预训练（Wang 等人，2020d）提出使用 ASR 数据预训练 Seq2Seq 模型以习得转写能力。作者指出，若进一步通过基于帧的掩码语言建模学习语义概念，以及通过基于帧的双语词汇翻译学习词对齐，可显著提升语音翻译性能。现有端到端模型通常仅预训练编码器或采用多任务学习，导致编码器无法区分转写、语义概念学习与对齐这三个本应分工明确的任务，实验数据验证了这一理论假设。听-理解-翻译模型（LUT）（Dong 等人，2021）在 Seq2Seq 框架中引入外部 ASR 损失函数，其核心创新在于构建语义编码网络——该网络通过转写编码器输出与目标文本 BERT 嵌入之间的均方误差损失，实现语义表征的优化。 这种策略隐式地构建并训练了一个用于翻译的神经机器翻译模型。在低资源场景下，使用自动语音识别和/或机器翻译进行预训练也被证明是有效的（Zhang 等人，2022a）。
2. 2.

   ST using MT: This section discusses approaches that use either MT data for pre-training or directly using a pre-trained MT model in the ST decoder. These approaches rely on the idea of generating pseudotext and then translating them using MT. For example, Unsupervised Term Discovery (UTD) (Bansal et al., 2017) groups repeated words into pseudo-text, which is subsequently used for training an MT model using the parallel pseudo-text and target translations. The main advantage of such a system is that it can translate some content words under low-resource settings. The overall results are not very promising on the Spanish-English Call-Home dataset. Another limitation of this work is that the approach is not an E2E in a true sense as it involves two models– a UTD and an MT model. A weakly supervised learning method for ST (Jia et al., 2019) that outperforms multi-task learning takes advantage of the pre-trained MT and TTS synthesis module. Pre-trained MT model is used as a teacher to guide the student ST model in (Liu et al., 2019) (such an approach is dubbed as knowledge distillation (KD)). They, however, rely on source language text and do not improve upon the pipeline system. Following along, (Gaido et al., 2020b) explores word, sentence and sequence-interpolation based KD approaches for transferring knowledge from pre-trained MT to ST model.

   Report issue for preceding element
   
   2. 基于机器翻译的语音翻译：本节探讨利用机器翻译数据进行预训练或直接在语音翻译解码器中使用预训练机器翻译模型的方法。这些方法的核心思想是先生成伪文本，再通过机器翻译进行转换。例如无监督术语发现（UTD）（Bansal 等人，2017）将重复词汇聚类为伪文本，随后利用伪文本与目标译文的平行语料训练机器翻译模型。该体系的主要优势在于能在低资源环境下翻译部分实义词。但在西班牙语-英语 Call-Home 数据集上的整体表现不尽如人意。该工作的另一局限在于其本质上并非真正的端到端系统，而是包含 UTD 和机器翻译两个独立模型。Jia 等人（2019）提出的弱监督语音翻译学习方法通过利用预训练机器翻译和文本转语音合成模块，其性能超越了多任务学习方案。Liu 等人（2019）采用预训练机器翻译模型作为教师模型指导学生语音翻译模型（该方法被称为知识蒸馏），但该方法依赖源语言文本且未超越传统级联系统的性能。 随后，(Gaido 等人，2020b)探索了基于单词、句子和序列插值的知识蒸馏方法，用于将预训练机器翻译模型的知识迁移至语音翻译模型。
3. 3.

   ST using both MT and ASR: This section discusses works employing MT and ASR pre-trained models (Bahar et al., 2020; Tsiamas et al., 2022a) or losses for transfer or multitask learning.

   Report issue for preceding element
   
   3. 结合机器翻译与自动语音识别的语音翻译：本节讨论利用机器翻译和自动语音识别预训练模型(Bahar 等人，2020；Tsiamas 等人，2022a)或损失函数进行迁移学习或多任务学习的研究工作。

   Multitask learning proves to be effective when CTC loss is combined with ASR and MT loss in (Bahar et al., 2019a) using various E2E ST architectures such as direct, multitask many-to-one, one-to-many, tied-cascade, and tied-triangle. They show that pre-trained models with ASR and MT losses achieve promising results. Contrary to claims of (Anastasopoulos and Chiang, 2018), tied-triangle architecture is no better than a direct model when fine-tuned properly. Since the ST task is similar to the MT task from the output perspective, works such as XSTNet (Ye et al., 2021) utilize external MT data to pre-train the encoder-decoder network extensively, then fine-tune it using parallel corpus data of MT, ST, ASR, and external MT data for optimizing the model using what they call progressive training. They achieve impressive performance on MuST-C and augmented Librispeech data. They also demonstrate improved performance on auxiliary tasks of MT and ASR. STPT model (Tang et al., 2022) proposes four sub-tasks for multitask pre-training: text-to-text (T2T), which is self-supervised; speech-to-phoneme which is supervised; acoustic learning, which is self-supervised, and ST which is supervised. Only T2T and ST tasks would subsequently be used for fine-tuning. Despite pre-training on “unlabeled” speech data, they obtained superior results on MuST-C data for the ST task. COSTT (Dong et al., 2020) pre-trains encoder using ASR data, the decoder using paired MT data, and then fine-tunes for the joint transcription-translation task. ComSL is a composite ST model relying on multitask learning with three losses ($L_{ASR},L_{MT},L_{ST}$) combined with cross-modality loss to bridge the gap (Le et al., 2023a). It is worth mentioning that ComSL does not require forced-align ST data and learns the cross-modality alignment during training. This however requires optimizing four different losses, viz. Masked Token Prediction, Speech to Text Mapping, Encoder Representation Matching, and Decoder Distribution Matching⁵⁵5please see (Le et al., 2023a) paper for more details. similar to (Tang et al., 2021b). Fused acoustic and text encoding-ST (FAT-ST) (Zheng et al., 2021b) follows the similar pre-training and fine-tuning idea as ComSL except that they propose to use any combination of training data from $D_{2^{\{u,x,v\}}}$ ⁶⁶6$2^{\{u,x,v\}}$ is the power set of triplets.. Essentially, they rely on masked language modeling (MLM) and translation language modeling (TLM) for pre-training (Conneau and Lample, 2019).
   多任务学习被证明是有效的，当(Bahar 等人，2019a)将 CTC 损失与自动语音识别(ASR)和机器翻译(MT)损失结合使用时，采用了多种端到端语音翻译架构，包括直接模型、多对一多任务、一对多多任务、级联绑定和三角绑定架构。研究表明，采用 ASR 和 MT 损失预训练的模型能取得优异效果。与(Anastasopoulos 和 Chiang，2018)的论断相反，经过适当微调后，三角绑定架构并不优于直接模型。由于从输出角度看语音翻译任务与机器翻译任务相似，诸如 XSTNet(Ye 等人，2021)等研究利用外部 MT 数据对编码器-解码器网络进行大规模预训练，继而采用 MT、ST、ASR 的平行语料库及外部 MT 数据进行渐进式训练优化模型。该模型在 MuST-C 和增强版 Librispeech 数据集上表现卓越，同时在 MT 和 ASR 辅助任务中也展现出性能提升。STPT 模型(Tang 等人，2022)提出了四种多任务预训练子任务：自监督的文本到文本(T2T)、有监督的语音到音素、自监督的声学学习以及有监督的语音翻译。 后续仅使用文本到文本（T2T）和语音转文本（ST）任务进行微调。尽管基于"未标注"语音数据进行预训练，他们在 MuST-C 数据集的 ST 任务中仍取得了优异表现。COSTT（Dong 等，2020）采用 ASR 数据预训练编码器，利用配对机器翻译数据预训练解码器，最后针对联合转录-翻译任务进行微调。ComSL 作为复合型 ST 模型，通过结合三重损失函数（ $L_{ASR},L_{MT},L_{ST}$ ）与跨模态损失来实现模态间隙的桥接（Le 等，2023a）。值得注意的是，ComSL 无需强制对齐的 ST 数据，而是在训练过程中自动学习跨模态对齐。但该方法需要同步优化四种不同损失函数：掩码标记预测、语音到文本映射、编码器表征匹配和解码器分布匹配 ⁵ ，其思路与（Tang 等，2021b）相似。融合声学与文本编码的 ST 模型（FAT-ST）（Zheng 等，2021b）沿用了与 ComSL 相似的预训练-微调框架，但其创新点在于允许任意组合使用来自 $D_{2^{\{u,x,v\}}}$ ⁶ 的训练数据。该模型本质上依赖于掩码语言建模（MLM）和翻译语言建模（TLM）进行预训练（Conneau 和 Lample，2019）。

   Report issue for preceding element
4. 4.

   Non-Autoregressive Modeling⁷⁷7 We present the discussion of NAR within the multitask learning framework because all NAR E2E ST models are optimized within the multitask framework. As discussed in the background section, an alternative approach to Autoregressive (AR) modeling is Non-Autoregressive (NAR) modeling. AR assumes that the output tokens are conditional dependent on the previously generated tokens. However, it causes significant latency during inference. NAR models solve this problem by outputting all the translated tokens in parallel thus speeding up the inference. Formally, they are given by (23)

   Report issue for preceding element

   $$p({\bf v}|{\bf u};\theta)=\prod_{t=1}^{T_{v}}p(v_{t}|,{\bf u};\theta)$$

   (23)

   There has been a surge in applying the non-autoregressive models (AR) in ASR and MT and it has prompted ST researchers to apply to it too. For example, (Inaguma et al., 2020a, 2021) trains NAR and autoregressive decoding conditioned on a shared speech encoder. Another line of NAR works (Chuang et al., 2021) explores CTC with ASR as an auxiliary task. CTC-based encoder only architecture ((Inaguma et al., 2020a, 2021) use encoder and decoder both) for NAR E2E ST task is shown to perform comparably or better than strong AR models in (Xu et al., 2023a).
   非自回归模型（NAR）在自动语音识别（ASR）和机器翻译（MT）领域的应用激增，这也促使语音翻译（ST）研究者开始采用该技术。例如，(Inaguma et al., 2020a, 2021) 训练了基于共享语音编码器的非自回归与自回归联合解码方案。另一类非自回归研究 (Chuang et al., 2021) 探索了以 ASR 作为辅助任务的 CTC 方法。研究表明，(Xu et al., 2023a) 提出的纯编码器架构（Inaguma 等人采用编码器-解码器架构）CTC 方案在端到端非自回归语音翻译任务中，其性能可媲美或超越强自回归模型。

   Report issue for preceding element
   
   4. 非自回归建模 ⁷ 如背景部分所述，自回归（AR）建模的替代方案是非自回归（NAR）建模。自回归模型假设输出标记依赖于先前生成的标记，但这会导致推理时产生显著延迟。非自回归模型通过并行输出所有翻译标记来解决该问题，从而加速推理过程。其形式化表达如公式(23)所示

Discussion: Our study of Seq2Seq-based frameworks for ST task reveals that (a) structural bias can be obtained by stacked/pyramidal RNN and alignment smoothing, (b) regularizers such as transitivity and invertibility improves Character Error Rate, (c) HLR helps in transcription as well as translation, and (d) changing the self-attention of the encoder with a logarithmic distance penalty enhances translation, (e) Progressive training needs a huge data and training time to achieve superior results, and (f) multitask pre-training can be used to leverage unlabeled speech data. (Zhang et al., 2022a) shows that ST models trained from scratch using only ST tasks perform on par with or surpass pre-trained models. To achieve such results, proposed best practices include a smaller vocabulary, a wider feedforward layer, a deep speech encoder with the post-layer norm, CTC-based regularization, and parameter-distance penalty. Pre-training is still useful in low-resource data regimes. Transferring knowledge via KD from pre-trained MT to ST causes gender bias, omission of sentences, and generic verbal-tense choice. Use of large vocabulary and models is effective for NAR E2E ST task (Inaguma et al., 2020a). It indicates that leveraging NAR with LLMs may be a future direction to explore.
讨论：我们对基于 Seq2Seq 的语音翻译（ST）任务框架的研究表明：(a) 通过堆叠/金字塔式 RNN 和对齐平滑可获得结构偏置；(b) 传递性和可逆性等正则化器能改善字符错误率；(c) 分层线性回归（HLR）对转写和翻译均有助益；(d) 将编码器的自注意力机制改为带对数距离惩罚的形式可提升翻译质量；(e) 渐进式训练需要大量数据和训练时间才能获得优异结果；(f) 多任务预训练可利用未标注语音数据。Zhang 等人(2022a)的研究显示，仅使用 ST 任务从头训练的模型性能可媲美或超越预训练模型。实现该效果的最佳实践包括：采用更小的词表、更宽的前馈层、带后层归一化的深度语音编码器、基于 CTC 的正则化以及参数距离惩罚。在低资源数据场景下，预训练仍具价值。通过知识蒸馏（KD）从预训练机器翻译模型迁移知识会导致性别偏见、句子遗漏和通用时态选择问题。 使用大规模词汇和模型对非自回归端到端语音翻译任务具有显著效果（Inaguma 等人，2020a）。这表明将非自回归模型与 LLMs 结合可能是未来值得探索的研究方向。

Streaming frameworks for ST tasks start outputting target tokens on seeing only partial inputs, that is, the translation of the input as soon as it arrives without waiting for the entire input. They are also known as Simultaneous ST (SimulST or SST)⁸⁸8Note that in MT literature, some works such as (Iranzo-S’anchez et al., 2022) differentiate between Streaming and Simultaneous setting where sentences are treated independently from each other. However, in ST, we find that existing works make no differentiation between them. (Goldman-Eisler, 1972; Fügen et al., 2007; Tsiartas et al., 2013; Grissom II et al., 2014). It finds application in online speech translation and video dubbing, to name a few. Traditionally, the streaming ST problem has been solved by feeding the segmented output of a streaming ASR model to a streaming MT model (Oda et al., 2014; Iranzo-Sánchez et al., 2020). However, due to the cascade nature of the model, it is prone to high latency and error propagation (Arivazhagan et al., 2019b, 2020; Zaidi et al., 2022). The SST problem faces several issues in practical implementation; reordering, acoustic ambiguity, and variable speech rate, and long inputs being prominent among them. Our literature survey reveals that most of the existing works focus on handling long streaming inputs and therefore, the discussion underneath revolves around that. Other issues mentioned above may also be considered for designing practical SST models.
面向语音翻译任务的流式处理框架在仅接收部分输入时便开始输出目标语符，即对输入内容进行实时翻译而无需等待完整输入。这类技术亦被称为同步语音翻译（SimulST 或 SST） ⁸ （Goldman-Eisler，1972；Fügen 等，2007；Tsiartas 等，2013；Grissom II 等，2014），其典型应用场景包括在线语音翻译和视频配音等。传统解决方案通常将流式语音识别模型的片段输出馈送至流式机器翻译模型进行处理（Oda 等，2014；Iranzo-Sánchez 等，2020）。然而由于级联模型的结构特性，该方法存在高延迟和错误传播的固有缺陷（Arivazhagan 等，2019b，2020；Zaidi 等，2022）。同步语音翻译在实际应用中面临若干挑战，其中词序重组、声学歧义、语音速率变化及长序列输入等问题尤为突出。文献调研表明，现有研究多聚焦于处理长流式输入，故下文讨论将围绕该主题展开。其他提及的问题在设计实用化同步翻译模型时亦需纳入考量。

Existing streaming frameworks intervene Seq2Seq framework at various places to design SST models. These are (a) encoder-level, (b) decoder-level, and (c) input/latent-level.
现有流式框架通过在 Seq2Seq 架构的不同环节进行干预来设计语音到文本翻译模型。具体包括：(a)编码器层面，(b)解码器层面，以及(c)输入/隐层层面。

1. 1.

   Encoder-level: SOTA SST models use transformers as encoders. Due to the self-attention operation which looks at the entire utterance, it is unsuitable for streaming inputs. There exist some works that design encoders specialized for streaming inputs. For example, augmented memory transformer (Wu et al., 2020; Ma et al., 2020c) splits the utterance $U$ into smaller-segments $S=[s_{1},\ldots]$. Each segment $s_{n}$ consists of left context $I_{n}$, main context $c_{n}$, and right context $r_{n}$. Self-attention is calculated at the segment level only thereby reducing the time complexity. Augmented memory propagates the information from one segment to the other. Incremental transformer (Zhang et al., 2020) leverages a unidirectional encoder based on unidirectional-attention with future context masked for handling streaming inputs.

   Report issue for preceding element
   
   1. 编码器层面：当前最先进的语音到文本翻译模型采用 Transformer 作为编码器。由于自注意力机制需要处理完整话语，这种架构不适合流式输入。已有研究设计了专门针对流式输入的编码器。例如增强记忆 Transformer（Wu 等人，2020 年；Ma 等人，2020c 年）将话语 $U$ 分割为较小片段 $S=[s_{1},\ldots]$ ，每个片段 $s_{n}$ 包含左上下文 $I_{n}$ 、主上下文 $c_{n}$ 和右上下文 $r_{n}$ 。自注意力仅在片段层面计算，从而降低时间复杂度。增强记忆机制实现了片段间的信息传递。增量 Transformer（Zhang 等人，2020 年）则采用基于单向注意力的编码器，通过掩码未来上下文来处理流式输入。
2. 2.

   Decoder-level: Instead of modifying encoders, some works such as (Dalvi et al., 2018; Liu et al., 2020a; Nguyen et al., 2021; Guo et al., 2024) propose incremental decoding (see fig. 7). In this framework, input speech is divided into fixed-size chunks and decoded every time a new chunk arrives. To avoid distractions from constantly changing hypotheses, selected chunk-level predictions are committed to and no longer modified. The decoding of the next chunk is conditioned by the predictions committed. Instead of conditioning on all chunk-level predictions, a prefix function is chosen to select a partial hypothesis because early chunks contain limited information (Liu et al., 2020a). There exist several strategies for choosing the prefix function. For example, Hold-$n$ and LA-$n$ (Liu et al., 2020a), SP-$n$ (Nguyen et al., 2021) and Regularized Batched Inputs (R-BI) (Guo et al., 2024). Of these, Hold-$n$ either withholds or deletes the last $n$ tokens in each chunk, LA–$n$ involves displaying the agreeing prefixes of $n$ consecutive chunks. SP-$n$ stands for shared prefix of all best-ranked hypotheses. Contrary to these, RB-I applies various augmentations to input chunks to achieve regularization and SOTA results on the IWSLT SimulST task.

   Report issue for preceding element
   
   2. 解码器层面：与修改编码器不同，部分研究（Dalvi 等人，2018；Liu 等人，2020a；Nguyen 等人，2021；Guo 等人，2024）提出了增量式解码方案（见图 7）。该框架将输入语音划分为固定大小的片段，每当新片段到达时进行解码。为避免持续变化的假设干扰，已选定的片段级预测结果将被固定且不再修改。后续片段的解码过程以已确认的预测结果为条件。由于早期片段包含信息有限（Liu 等人，2020a），解码过程并非基于所有片段级预测，而是通过前缀函数选择部分假设。现有多种前缀函数选择策略，例如 Hold- $n$ 和 LA- $n$ （Liu 等人，2020a）、SP- $n$ （Nguyen 等人，2021）以及正则化批量输入（R-BI）（Guo 等人，2024）。其中 Hold- $n$ 策略对每个片段末尾的 $n$ 个词元进行保留或删除处理，LA- $n$ 策略需显示 $n$ 个连续片段的一致前缀，SP- $n$ 则表示所有最优假设的共享前缀。 与此相反，RB-I 通过对输入块应用多种数据增强技术来实现正则化，并在 IWSLT 同声传译任务上取得了当前最优性能。
3. 3.

   Input/latent-level: Since speech input is too fine-grained, deciding when to READ and WRITE is challenging. The existing works introduce pre-decision module which segments the input speech at fixed-chunks (fixed) or word-boundary (flexible). Similarly, READ/WRITE policy can be fixed or adaptive (Ma et al., 2020b). Most research in SST concentrates on either improving speech encoding or pre-decision while relying on fixed policies such as wait-$k$. In this section, we discuss fixed and adaptive pre-decisions/policies. These techniques are combined with Seq2Seq frameworks to devise streaming ST models.
   输入/潜在层面：由于语音输入过于细粒度，决定何时读取与写入具有挑战性。现有研究引入了预决策模块，将输入语音按固定分块（固定式）或词边界（灵活式）进行分割。类似地，读取/写入策略可采用固定或自适应方式（Ma 等，2020b）。当前语音到文本翻译研究主要集中于改进语音编码或预决策模块，同时依赖固定策略如 wait- $k$ 。本节将探讨固定式与自适应式预决策/策略，这些技术与序列到序列框架相结合，可构建流式语音翻译模型。

   Report issue for preceding element

   

   Report issue for preceding element

   

   Report issue for preceding element

   Report issue for preceding element

   Wait-$k$ policy (Ma et al., 2018) (shown in fig. 9) learns the parameters $\theta$ of the model by optimizing the negative log-likelihood $-\sum_{(\mathbf{u,v})\in D}\log p(\mathbf{v}|\mathbf{u};k;\theta)$, where $k$ is the number of segments to look before starting translation (see Fig. 9). The probability $p(\cdot)$ is calculated as
   等待- $k$ 策略（Ma 等人，2018 年）（如图 9 所示）通过优化负对数似然 $-\sum_{(\mathbf{u,v})\in D}\log p(\mathbf{v}|\mathbf{u};k;\theta)$ 来学习模型参数 $\theta$ ，其中 $k$ 表示开始翻译前需要观察的语音片段数量（参见图 9）。概率 $p(\cdot)$ 的计算公式为：

   Report issue for preceding element

   $$p({\bf v}|{\bf u};k;\theta)=\prod_{t=1}^{T_{v}}p(v_{t}|v_{<t},u_{t+k};\theta)$$

   (24)

   wait-$k$ policy guarantees that the model can look at $t+k-1$ speech segments while predicting token $v_{t}$ (Ren et al., 2020). However, one limitation of the wait-$k$ policy is that it fails to do a beam search while decoding except for long-tail (Ma et al., 2018). To solve this problem, (Zeng et al., 2021) proposes a wait-$k$ stride-$N$ policy. It essentially is a wait-$k$ policy with the addition of $N$ READ and WRITE operations until the end of the sentence after reading the first $k$-segments. To determine the $k$-segments, (Chen et al., 2021b) leverages streaming ASR to guide the direct simultaneous ST decoding via beam search.
   等待- $k$ 策略确保模型在预测标记 $v_{t}$ 时能够观察 $t+k-1$ 个语音片段（Ren 等人，2020 年）。然而该策略的局限性在于，除长尾情况外（Ma 等人，2018 年），解码时无法执行束搜索。为解决此问题，（Zeng 等人，2021 年）提出了等待- $k$ 跨步- $N$ 策略，其本质是在读取前 $k$ 个片段后，通过添加 $N$ 个 READ 和 WRITE 操作直至句子结束的等待- $k$ 策略。（Chen 等人，2021b 年）则利用流式自动语音识别技术，通过束搜索确定 $k$ 个片段来指导直接同步语音翻译解码。

   Report issue for preceding element

   

   Report issue for preceding element

   As discussed above, determining when to write is crucial for efficient SST. Contrary to wait-$k$ policy which is a fixed-policy, Segmentation can be performed on the embedded speech using CTC (Ren et al., 2020), attention mechanism (Papi et al., 2022b), or incremental BEAM search (Yan et al., 2023). Essentially, these works adapt offline ST to SST showing spectacular performance on benchmark datasets. Note that the models proposed in (Papi et al., 2022b; Yan et al., 2023) train models in a cascade manner while the inference is E2E. Another issue with the fixed policy is that the model can not speed up or slow down appropriately with the input types. Other examples of fixed-policy are Wait-If* (Cho and Esipova, 2016b) and Monotonic Chunkwise Attention (MoChA) (Chiu and Raffel, 2018) that has been used in simultaneous MT and may be explored for SST.
   如上所述，确定何时书写对于高效的语音到文本同传至关重要。与固定策略的 Wait- $k$ 策略不同，分段操作可通过连接时序分类（Ren 等，2020）、注意力机制（Papi 等，2022b）或增量束搜索（Yan 等，2023）在嵌入式语音上实现。这些研究本质上将离线语音翻译技术适配为同传场景，在基准数据集上展现出卓越性能。需注意的是，（Papi 等，2022b；Yan 等，2023）提出的模型采用级联方式训练，而推理过程则是端到端的。固定策略的另一缺陷在于模型无法根据输入类型动态调节处理速度。其他固定策略实例包括 Wait-If*（Cho 和 Esipova，2016b）及单调分块注意力机制（MoChA）（Chiu 和 Raffel，2018），这些方法已在同步机器翻译中应用，或可探索其在语音到文本同传中的适用性。

   Report issue for preceding element

   The works mentioned above require that encoded speech be segmented so that the decoder can apply the wait-$k$ policy. The goal of segmentation is to identify the word, sub-word, or phone boundary which are usually not even (due to silences, longer syllables, etc.). That means the number of acoustic units varies with time in each segment. Monotonic-segmented Streaming ST (MoSST) (Dong et al., 2022) is based on learning when to translate, which has a monotonic segmentation module located between the acoustic encoder and the transformer. It has an Integrate-and-Fire (IF) neuron (Abbott, 1999), which fires above a threshold when the context is developed. If the context is not developed, the neuron receives signals and accumulates the acoustic vectors, thus mimicking adaptive policy for READ-WRITE operation. IF strategy has shown impressive performance in simultaneous ASR (Dong and Xu, 2019) and ST (Chang and yi Lee, 2022). It can be used for monotonic segmentation of the speech input along with adaptive decision strategy (Dong et al., 2022). Another adaptive policy-based technique is Monotonic Infinite Lookback Attention (MILk) (Arivazhagan et al., 2019b) used in simultaneous MT can be explored for SST. It essentially is a Monotonic Attention mechanism (Raffel et al., 2017) that extends to infinite encoder states, theoretically, in the past and trains the MT model along with the MILk. It achieves better quality-latency trade-offs than MoCHA thanks to its soft attention to all the encoder states and hard attention. Monotonic Multihead Attention (MMA) (Ma et al., 2019) that extends MILK to multiple heads has been used for SST by (Ma et al., 2020b). Its variants Efficient MMA (Ma et al., 2023) solve numerical stability and biased monotonic alignment issues present in MMA but have not been explored for SST tasks. Adaptive segmentation based on an adaptive policy that takes into account acoustic features and translation history (called meaningful units) is another effective mechanism for SST (Zhang et al., 2022b).
   上述研究要求对编码后的语音进行分段，以便解码器能够应用等待- $k$ 策略。分段的目的是识别通常不均匀的词、子词或音素边界（由静音、较长音节等因素导致），这意味着每个片段中的声学单元数量随时间变化。基于学习翻译时机的单调分段流式语音翻译（MoSST）（Dong 等人，2022）在声学编码器与 Transformer 之间设置了单调分段模块，其采用具有积分发放特性的神经元（Abbott，1999），当上下文信息充分时会触发阈值以上的发放。若上下文未充分发展，该神经元将持续接收信号并累积声学向量，从而模拟读写操作的自适应策略。该积分发放策略在同步语音识别（Dong 和 Xu，2019）及语音翻译（Chang 和 yi Lee，2022）中展现出卓越性能，可结合自适应决策策略（Dong 等人，2022）实现语音输入的单调分段处理。 另一种基于自适应策略的技术是单调无限回望注意力机制（MILk）（Arivazhagan 等人，2019b），该技术最初用于同步机器翻译，也可探索应用于语音到文本翻译。其本质是一种单调注意力机制（Raffel 等人，2017）的理论延伸，能够无限回溯编码器历史状态，并与 MILk 联合训练机器翻译模型。由于其对所有编码器状态采用软注意力与硬注意力相结合的方式，相比 MoCHA 实现了更优的质量-延迟权衡。单调多头注意力（MMA）（Ma 等人，2019）将 MILk 扩展至多头机制，已被（Ma 等人，2020b）应用于语音到文本翻译。其改进版本高效 MMA（Ma 等人，2023）解决了原 MMA 中存在的数值稳定性和单调对齐偏差问题，但尚未在语音到文本翻译任务中得到探索。另一种有效机制是基于声学特征和翻译历史（称为意义单元）的自适应策略进行分段处理（Zhang 等人，2022b）。

   Report issue for preceding element

   Both fixed and adaptive policy mechanisms employ segmentation modules that are outside the translation module. As such, it breaks the acoustic integrity and potentially may drop the translation performance. Therefore, efforts such as (Zhang and Feng, 2023) propose differentiable segmentation (DiSeg) learned jointly with the translation model using expectation training. DiSeg essentially predicts a Bernoulli random variable $\sigma(FFN(u_{i}))$, via a feed-forward network (FFN), to decide when to segment. After segmentation, they apply segmented attention which combines unidirectional and bidirectional attention into one while masking future speech frames. Expectation training first constrains the number of segments followed by learning segmentation from the translation model both at semantic and acoustic levels (Zhang and Feng, 2023).
   固定策略与自适应策略机制均采用独立于翻译模块的分割组件，这破坏了语音信号的完整性并可能导致翻译性能下降。为此，张与冯（2023）提出了可微分分割（DiSeg）方法，通过期望训练实现与翻译模型的联合学习。DiSeg 本质上通过前馈神经网络预测伯努利随机变量 $\sigma(FFN(u_{i}))$ 以决策分割时机，随后采用融合单向与双向注意力的分段注意力机制，同时屏蔽未来语音帧。期望训练首先约束分段数量，继而从语义和声学两个层面实现翻译模型驱动的分割学习（Zhang and Feng, 2023）。

   Report issue for preceding element

The discussion so far covered encoder, and decoder level changes, and fixed and adaptive policies used for segmentation, to develop SST models within the Seq2Seq frameworks. Another way to design SST models is by Transduction. It is the process of mapping a sequence to another sequence (Jurafsky and Martin, 2008). A transducer is a special type of Seq2Seq model that solves a few inherent problems. For example, online processing of the long inputs and monotonic sequence alignment is the biggest problem with Seq2Seq models (Graves, 2012), solved by transducers. Below we discuss a special type of transducer called RNN-T and its improvements.
前文已涵盖在序列到序列（Seq2Seq）框架下构建语音到文本翻译（SST）模型时，编码器与解码器层级的改进，以及用于分段的固定策略与自适应策略。另一种设计 SST 模型的方法是通过转导（Transduction）实现。转导是将一个序列映射为另一个序列的过程（Jurafsky 和 Martin，2008）。转导器作为 Seq2Seq 模型的特例，能解决若干固有问题。例如，Seq2Seq 模型在处理长输入流的在线运算与单调序列对齐方面存在显著缺陷（Graves，2012），而转导器可有效解决这些问题。下文将重点讨论 RNN-T 这一特殊转导器及其改进方案。

RNN-T is a transducer that can learn alignment between two sequences in an online/streaming fashion (Graves, 2012) as shown in fig. 10 (a). Formally, it learns the conditional probability $p({\bf v|u})$ by marginalizing all possible alignment paths $A(\bf u,v)$ including blank symbol $\phi$ as
RNN-T 是一种能够以在线/流式方式学习两个序列间对齐关系的转导器（Graves，2012），如图 10(a)所示。其形式化定义为通过边缘化所有可能对齐路径（包含空白符号）来计算条件概率。

RNN-T differs from Seq2Seq in the sense that it divides the decoder into a predictor and a joiner. The predictor takes the previous time step output and yields the representation to be consumed by the joiner along with the hidden representation of the input from the encoder. Since the predictor does not look at the input, it can be pre-trained on the text-only data in a low-data scenario. There have been several SST models proposed based on variants of RNN-T which we discuss next.
RNN-T 与 Seq2Seq 的区别在于它将解码器拆分为预测器和连接器。预测器接收前一时刻输出并生成表征，该表征将与编码器输入的隐藏表征共同输入连接器。由于预测器不直接观察输入数据，在低数据场景下可仅通过文本数据进行预训练。目前已提出多种基于 RNN-T 变体的端到端语音翻译模型，我们将在下文详述。

One of the main issues with RNN-T is the strict monotonic alignment between the input and output sequences which makes them unsuitable for tasks requiring reordering such as MT, ST, etc. For example, Cross-Attention Augmented Transducer (CAAT shown in fig. 10(b)) optimizes translation and policy models in tandem (Liu et al., 2021a). It eliminates the RNN-T’s strict monotonic restriction for reordering in the translation. Using transformers as encoders to reduce the multi-step memory footprint causes a significant delay for CAAT. The use of regularization terms and substantial hyperparameter adjustment are some other limitations of CAAT. An extension of it in (Xue et al., 2022) leverages Transformer Transducer (TT) networks with attention pooling for streaming E2E ST tasks. Attention divides the input audio into chunks of specific sizes. At any time, processing any input frame $\bf u_{t}$ can only see frames within its chunk and a fixed number of left chunks. By sharing the encoder, they also propose a variant to handle E2E ST tasks in multilingual settings. The adaptive READ and WRITE policy choices between encoder output and ground truth contribute to its success. The same authors (Wang et al., 2023) propose to combine the benefits of language-specific and language-agnostic encoders within the TT framework. A shared encoder takes LIDs as gating values and computes weights for each language through the source LID scheduling scheme. The empirical results demonstrate superior performance and a smaller number of trainable parameters than bilingual ST. Adaptive (dynamic) policy for segmenting speech input has recently been explored in a Seq2Seq transduction setting by (Tan et al., 2024). It essentially applies a cross-attention mechanism to decide when to segment the input followed by dynamic compression via anchor representation. Thus, it saves memory and achieves a better latency-quality trade-off.
RNN-T 的主要问题之一在于输入与输出序列间严格的单调对齐特性，这使得其不适用于需要重排序的任务（如机器翻译、语音翻译等）。以交叉注意力增强传感器（CAAT，见图 10(b)）为例（Liu 等人，2021a），该模型通过同步优化翻译策略模块与策略模型，解除了 RNN-T 对翻译过程中重排序的严格单调限制。但采用 Transformer 编码器来降低多步内存占用的设计，导致 CAAT 存在显著延迟。此外，正则化项的使用与大量超参数调整也是 CAAT 的局限性所在。Xue 等人（2022）提出的改进方案采用带注意力池化的 Transformer 传感器（TT）网络处理流式端到端语音翻译任务，通过将输入音频分割为特定大小的块，使任意时刻处理的输入帧 $\bf u_{t}$ 仅能感知当前块及固定数量左侧块内的帧。该研究还通过共享编码器提出多语言端到端语音翻译的变体方案，其成功关键在于编码器输出与真实标签间自适应的 READ-WRITE 策略选择机制。 同一批作者（Wang 等，2023）提出在 TT 框架内结合语言特定编码器与语言无关编码器的优势。其共享编码器将语言标识符作为门控值，通过源语言标识符调度方案计算各语言的权重。实证结果表明，该方法不仅性能优越，且可训练参数数量少于双语语音翻译系统。近期（Tan 等，2024）在序列到序列转换框架中探索了语音输入分段的动态自适应策略，该方法通过交叉注意力机制决策分段时机，并采用锚点表示进行动态压缩，从而节省内存并实现更优的延迟-质量平衡。

Besides Transducer and Seq2Seq models, re-translation is another approach adapted for SST task by (Niehues et al., 2016, 2018; Arivazhagan et al., 2019a, 2020) though in a cascade setting. In this approach, the translated output can be re-generated after a fixed amount of time and displayed later for better quality. Though it reduces latency by being greedy to display the partial translation, the output is highly unstable and causes flickring effect. This may give rise to a bad user experience. To mitigate instability, (Arivazhagan et al., 2020) propose a metric called erasure which takes into the length of the suffix deleted during re-translation. Dynamic masking of MT output in a cascade of streaming ASR and MT for improving stability has been explored in (Yao and Haddow, 2020). Another approach to reducing instability is luminance contrast and the Discrete Fourier Transform used in (Liu et al., 2023).
除 Transducer 和 Seq2Seq 模型外，重翻译是(Niehues 等, 2016, 2018; Arivazhagan 等, 2019a, 2020)在级联框架下采用的另一种语音到文本翻译方法。该方法可在固定时间间隔后重新生成翻译结果并延迟显示以提升质量。虽然通过贪心策略显示部分翻译降低了延迟，但输出极不稳定且会产生闪烁效应，可能导致较差的用户体验。为缓解不稳定性，(Arivazhagan 等, 2020)提出了考虑重翻译过程中删除后缀长度的"擦除"度量指标。(Yao 和 Haddow, 2020)探索了在流式语音识别与机器翻译级联系统中动态掩码机器翻译输出的方法以提升稳定性。(Liu 等, 2023)则采用亮度对比度和离散傅里叶变换作为另一种降低不稳定性的解决方案。

Evaluation of SST models: SST models in the literature have been evaluated using the quality and latency metrics presented in §3. Often showing a trade-off between quality and latency. Most of the existing works attempt to balance the quality and latency ignoring the visualization and cognitive load on the viewer when displayed on a screen. Towards this end, (Papi et al., 2021b) emphasizes considering visualization as a metric to be evaluated along with the latency and quality. However, little effort has been made in this direction by the SST community. Therefore, we wish to draw the researcher’s attention to also consider visualization as an evaluation metric for SST models. Towards this end, (Liu et al., 2023) propose tokenized alignment, word updates with semantic similarity, and smooth animation of live captions. They find that it leads to a reduction in fatigue, and distractions while increasing the viewer’s reading comfort.
端到端语音转文本翻译模型的评估：现有文献中对 SST 模型的评估主要采用第 3 节所述的质量与延迟指标，这两者往往呈现此消彼长的关系。当前大多数研究试图平衡质量与延迟，却忽略了字幕在屏幕上显示时的可视化效果及观看者的认知负荷。为此，(Papi 等人，2021b)强调应将可视化作为与延迟和质量并列的评估指标。然而 SST 领域在此方向的探索仍显不足。因此我们呼吁研究者将可视化纳入 SST 模型的评估体系。(Liu 等人，2023)提出的分词对齐、基于语义相似度的单词更新以及实时字幕平滑动画技术，经证实能有效降低观看疲劳与注意力分散，同时提升阅读舒适度。

Discussion: SST is a challenging problem and in that, E2E SST poses a further impediment. Our findings suggest that using adaptive policy significantly improves the latency-quality trade-off. Learned policy mechanisms have been an ongoing research and adapting them for true long-form SST may open new possibilities. Exploring differentiable segmentation for long sequences is still tapped and requires more investigation. Re-translation is found to be on par with or better than SOTA streaming models (Arivazhagan et al., 2020) under a very low revision rate. Such a finding alludes to considering re-translation in an E2E SST system design.
讨论：语音到文本翻译（SST）本身已是一个具有挑战性的问题，而端到端 SST 则带来了更多障碍。我们的研究表明，采用自适应策略能显著改善延迟与质量的权衡关系。学习型策略机制一直是持续研究的课题，将其适配于真正的长文本 SST 可能开辟新的可能性。针对长序列的可微分分割研究仍处于探索阶段，需要更多深入调查。在极低修正率条件下，重翻译表现与当前最优流式模型相当或更优（Arivazhagan 等人，2020），这一发现暗示在端到端 SST 系统设计中应考虑引入重翻译机制。

A low-resource language (LRL) is one where speech and/or text data are scarcely available – usually not enough to pre-train Seq2Seq models. As such, LRLs present challenges of their own such as overfitting and poor generalization. This section will discuss works where ST models are developed especially for low-resource languages. The proposed models under this category have generic architecture as shown in Fig.11(a) which is similar to Seq2Seq ST models. We find the approaches mainly use pre-training the encoder on high-resource ASR data and subsequent fine-tuning on ST data. Another approach that has emerged in recent years to tackle LRL issues is SSL. For example, (Bansal et al., 2019) empirically demonstrates 100% performance improvement on ST tasks. They find that if the ASR language differs from the source and target languages, then pre-training on ASR data enhances ST task performance. Though the BLEU score is improved, the absolute BLEU score is only 7.1. In (Wang et al., 2022), the unsupervised ST is implemented for low-resource settings using pseudo-labels from unsupervised cascade models. SSL with discrete-speech unit (DSU) has been used to fine-tune the ST model on limited ST data (Lam et al., 2024).
低资源语言(LRL)是指语音和/或文本数据极其匮乏的语言——通常不足以预训练序列到序列模型。因此，低资源语言面临特有的挑战，如过拟合和泛化能力差。本节将重点讨论专门针对低资源语言开发的语音翻译模型。该类模型采用图 11(a)所示的通用架构，与常规序列到序列语音翻译模型类似。研究发现，这些方法主要通过对编码器进行高资源语音识别数据的预训练，再在语音翻译数据上进行微调。近年来兴起的另一解决方案是自监督学习(SSL)。例如(Bansal 等人，2019)通过实验证明该方法可使语音翻译任务性能提升 100%。他们发现，当语音识别训练语言与源语言/目标语言不同时，基于语音识别数据的预训练能显著提升翻译性能。虽然 BLEU 分数有所提高，但绝对值仅为 7.1 分。(Wang 等人，2022)则利用无监督级联模型生成的伪标签，实现了低资源环境下的无监督语音翻译。 采用离散语音单元（DSU）的自监督学习（SSL）方法已被用于在有限语音翻译数据上对 ST 模型进行微调（Lam 等人，2024 年）。

Code-mix language refers to speech where one primary language is used, but words or phrases from other (embedded) languages are also included. This phenomenon arises from a multitude of challenges, encompassing ambiguous vocabulary, fluctuating lexical representations, intermingling of languages at the word level, redundancy, and alterations in word sequencing. Therefore, it is non-trivial to handle code-mixing while building ST models.
语码混合语言指以某种主要语言为基础，同时夹杂其他（嵌入）语言词汇或短语的语音现象。该现象衍生出多重挑战，包括词汇歧义、词形表征波动、词汇层面的语言混杂、冗余信息以及词序变异等问题。因此，在构建语音翻译模型时处理语码混合现象具有显著难度。

We find that there exist only a few works on code-mix ST. In (Weller et al., 2022), the code-mix dataset is created with the existing publicly available corpora Fisher (Cieri et al., 2004) and Miami⁹⁹9https://github.com/apple/ml-code-switched-speech-translation. As shown in Fig. 11(b), code-mix ST models feed language ID in addition to speech input to the encoder of the Seq2Seq model (Weller et al., 2022). The Wav2Vec 2.0, an acoustic encoder, and mBART, a multilingual decoder, are used for both languages with an attention layer applied for the embedded language. The use of multilingual encoders and decoders is a common practice while building code-mix ST models (Yang et al., 2023). In particular, self-supervised multilingual pre-training with adapters may be explored further.
我们发现目前关于混合编码语音翻译（code-mix ST）的研究成果较少。Weller 等人（2022）利用现有公开语料库 Fisher（Cieri 等人，2004）和 Miami ⁹ 构建了混合编码数据集。如图 11(b)所示，混合编码语音翻译模型除了语音输入外，还需向序列到序列模型的编码器输入语言标识（Weller 等人，2022）。该研究采用声学编码器 Wav2Vec 2.0 和多语言解码器 mBART 处理双语数据，并通过注意力机制处理嵌入语言特征。使用多语言编码器-解码器架构是构建混合编码语音翻译模型的常见方法（Yang 等人，2023）。特别是基于适配器的自监督多语言预训练方法值得进一步探索。

There is an abundance of unlabeled speech and text data. Since manual annotation and creating a parallel corpus is costly, the natural instinct is to exploit unlabeled data for training ST models. This section reviews works where researchers make use of the unlabeled speech data to advance the ST task performance.
存在大量未标注的语音和文本数据。由于人工标注和创建平行语料库成本高昂，研究者很自然地会尝试利用未标注数据来训练语音翻译模型。本节综述了研究人员利用未标注语音数据提升语音翻译任务性能的相关工作。

For unsupervised ST tasks, it is common to leverage large-scale self-supervised and semi-supervised learning. For example, speech encoders such as Wav2vec 2.0 have been pre-trained in a self-supervised manner on Librilight data (Kahn et al., 2019) and used by (Li et al., 2020; Wang et al., 2021b) whereas the decoder is randomly initialized. The entire model is optimized on CoVoST 2 ST data, and the encoder is frozen. Thereby, self-training is executed to generate pseudo-labels for Libri-light data. The Wav2Vec 2.0 is a “student” model which is fine-tuned with ground truth CoVoST 2 data and pseudo labels. Finally, a language model (LM) is trained on CommonCrawl data and combined with the ST model to generate text via beam-search decoding. Following along, for training the E2E model, (Wang et al., 2021b) produces pseudo-labels by cascading ASR, text de-normalization, and MT in an Unsupervised manner. Wav2Vec 2.0 and mBART are optimized for domain adaption using in-domain data (Li et al., 2020). According to experimental results, the proposed method is effective for E2E models without pre-training. However, between supervised and unsupervised pre-trained models, performance gap is encountered, which may be investigated in future works.
在无监督语音翻译任务中，通常采用大规模自监督与半监督学习方法。例如，Wav2vec 2.0 等语音编码器通过自监督方式在 Librilight 数据集上进行预训练（Kahn 等人，2019），并被 Li 等人（2020）与 Wang 等人（2021b）采用，而解码器则采用随机初始化。整个模型基于 CoVoST 2 语音翻译数据进行优化时，编码器保持冻结状态。随后通过自训练为 Libri-light 数据生成伪标签。Wav2Vec 2.0 作为"学生"模型，使用 CoVoST 2 真实数据与伪标签进行微调。最终基于 CommonCrawl 数据训练语言模型（LM），与语音翻译模型结合通过束搜索解码生成文本。Wang 等人（2021b）进一步通过级联自动语音识别、文本规范化与机器翻译的无监督流程生成伪标签，用于端到端模型训练。Wav2Vec 2.0 与 mBART 通过领域内数据实现领域自适应优化（Li 等人，2020）。实验结果表明，该方法对无需预训练的端到端模型具有显著效果。 然而，在监督式与无监督式预训练模型之间仍存在性能差距，这值得在未来的研究中深入探讨。

The multilingual ST model aims to translate from/to multiple speech input/output languages. It can be one of many-to-one, one-to-many, or many-to-many. The ST models solve multilinguality issues using mainly three approaches: (a) language ID, (b) dual-decoder, and (c) pre-trained models.
多语言语音翻译模型旨在实现多种语音输入/输出语言之间的互译，其架构可分为多对一、一对多或多对多三种类型。当前语音翻译模型主要通过三种方法解决多语言问题：(a) 语言识别编码技术，(b) 双解码器架构，以及(c) 预训练模型应用。

1. 1.

   Language ID (LID) is the identification label that allows one to identify the target language and explicitly translate the speech simultaneously. The existing works handle multilinguality using LID either with encoder or decoder. In (Inaguma et al., 2019), the model uses LID in the decoder for one-to-many and many-to-many translation. They demonstrate impressive performance in translation from high-resource to low-resource languages without using any transcript data from LRL. However, using the LID embedding in the decoder (Gangi et al., 2019) is shown to underperform than using it in the encoder. The author shows that LID can be either concatenated or merged with the inputs and, when pre-trained with ASR data, can result in superior performance than the one-to-one system. The model, however, performs poorly when trained on many unrelated target languages. One-to-many and many-to-one multilingual ST systems of (Wang et al., 2020c, a) provide a good set of baselines for research purposes.

   Report issue for preceding element
   
   1. 语言识别标识（LID）是一种用于识别目标语言并实现语音同步显式翻译的标识标签。现有研究通过编码器或解码器结合 LID 处理多语言场景。Inaguma 等人（2019）提出的模型在解码器中采用 LID 实现一对多和多对多翻译，其研究表明：在不使用低资源语言转录数据的情况下，该模型能实现从高资源语言到低资源语言的出色翻译性能。然而 Gangi 等人（2019）证明，在解码器中使用 LID 嵌入的表现逊色于编码器方案。作者指出 LID 既可与输入向量拼接也可融合，当结合语音识别数据预训练时，其性能可超越单语翻译系统。但该模型在训练涉及多个不相关目标语言时表现欠佳。Wang 等人（2020c，a）提出的一对多与多对一多语言语音翻译系统为研究提供了良好的基准模型。
2. 2.

   Dual-decoder model is the transformer with two decoders, one for each ASR and ST, and the dual-attention mechanism. In (Le et al., 2020), a dual-decoder model is proposed to optimize it for ASR and ST tasks jointly. The author hypothesizes that a dual-attention mechanism can benefit each task by transferring knowledge instantly or in wait-$k$ policy mechanism. Their model generalizes earlier models proposed for one-to-many and bilingual ST models.

   Report issue for preceding element
   
   2. 双解码器模型是一种配备两个解码器的 Transformer 架构，分别用于语音识别（ASR）和语音翻译（ST）任务，并采用双重注意力机制。Le 等人（2020）提出通过联合优化方式使该模型同时适应 ASR 与 ST 任务。作者假设双重注意力机制能通过即时知识迁移或等待策略机制（wait- $k$ policy）使两项任务相互受益。该模型推广了早期针对一对多和双语语音翻译任务所提出的模型架构。
3. 3.

   Pre-trained Multilingual Models use a pre-trained encoder and decoder for acoustic modeling and language modeling, respectively. In (Li et al., 2020; Tran et al., 2020), the author shows that efficiently fine-tuning mBART, which is a pre-trained multilingual decoder (Liu et al., 2020c) can achieve SOTA results on CoVoST data on zero-shot cross-lingual and multilingual translation tasks. Along similar lines, (Le et al., 2021) shows that inserting adapters in between layers of the encoder-decoder framework and tuning them can improve the ST task performance over bilingual ST models. SeamlessM4T (Barrault et al., 2023), Whisper (Radford et al., 2023), and other foundation models are built using many of these concepts like language ID in the decoder, multilingual, multimodal, and multitask pre-training.

   Report issue for preceding element
   
   3. 预训练多语言模型分别采用预训练编码器与解码器进行声学建模和语言建模。Li 等人(2020)与 Tran 等人(2020)的研究表明，通过高效微调 mBART（一种预训练多语言解码器，Liu 等人 2020c 提出）可在 CoVoST 数据集上实现零样本跨语言及多语言翻译任务的 SOTA 性能。类似地，Le 等人(2021)证实，在编码器-解码器框架各层间插入适配器并进行调优，可超越双语语音翻译模型的性能表现。SeamlessM4T（Barrault 等人 2023）、Whisper（Radford 等人 2023）等基础模型均融合了这些核心设计理念，包括解码器语言识别、多语言多模态多任务预训练等技术要素。