RECAP: Retrieval-Augmented Audio Captioning
《RECAP：检索增强型音频字幕生成》

Main Contributions. We propose RECAP, REtrieval-Augmented Audio CAPtioning, a simple and scalable solution to the aforementioned problems of domain shifts. Similar to other audio captioning systems in literature [5, 6, 7], RECAP is built on an audio encoder and a language decoder (GPT-2 in our setting). However, we introduce three novel changes: (1) Instead of employing an audio encoder pre-trained only on audio, we use CLAP [1] as our audio encoder. CLAP is pre-trained on audio-text pairs to learn the correspondence between audio and text by projecting them into a shared latent space. Thus, CLAP hidden state representations are better suited for captioning due to their enhanced linguistic comprehension. (2) We condition the audio for caption generation by introducing new cross-attention layers between CLAP and the GPT-2. (3) Finally, beyond just conditioning audio, we also condition a custom-constructed prompt for training and inference. We construct the prompt using the top-k captions most similar to the audio from a datastore retrieved using CLAP. We provide more details in Section 3.1. RECAP builds on retrieval-augmented generation (RAG) [8], which offers multiple advantages discussed further in Section 3. RECAP is lightweight, fast to train (as we only optimize the cross-attention layers), and can exploit any large text-caption-only datastore in a training-free fashion. We evaluate RECAP on two benchmark datasets, Clotho [3] and AudioCaps [2], and show that while being competitive to the state-of-the-art in in-domain settings, RECAP outperforms all baselines in out-of-domain settings by a large margin. Additionally, RECAP can effectively caption novel audio events never seen during training and can better generate captions for compositional audios with multiple audio events.
主要贡献。我们提出 RECAP（检索增强音频描述），针对上述领域偏移问题提供了一种简单且可扩展的解决方案。与文献中其他音频描述系统[5,6,7]类似，RECAP 基于音频编码器和语言解码器（本研究中采用 GPT-2）构建。但我们引入了三项创新改进：（1）采用 CLAP[1]作为音频编码器替代仅预训练于音频的编码器。CLAP 通过音频-文本对预训练，将两者映射至共享潜在空间以学习对应关系，其隐藏状态表征因增强的语言理解能力更适用于描述任务；（2）通过在 CLAP 与 GPT-2 间新增交叉注意力层，实现音频条件化描述生成；（3）除音频条件化外，我们还为训练和推理设计了定制化提示条件。该提示由 CLAP 从数据库中检索出的、与输入音频最相似的前 k 条描述构建而成。 我们将在第 3.1 节提供更多细节。RECAP 基于检索增强生成（RAG）[8]构建，该框架具有第 3 节将进一步讨论的多项优势。RECAP 具有轻量级、训练快速（因为我们仅优化交叉注意力层）的特点，并能以无需训练的方式利用任何仅含文本-字幕的大型数据存储。我们在 Clotho[3]和 AudioCaps[2]两个基准数据集上评估 RECAP，结果表明：在领域内设置中与最先进技术保持竞争力的同时，RECAP 在领域外设置中以显著优势超越所有基线方法。此外，RECAP 能有效为训练中从未见过的新音频事件生成描述，并能更好地为包含多个音频事件的组合音频生成字幕。

Automated Audio Captioning. Current work in audio captioning primarily employs encoder-decoder models where a caption is generated by an autoregressive language decoder conditioned on representations obtained from an audio encoder [5, 6, 7]. The language decoder employed is either pre-trained on web-scale data [5, 6, 7] or learned from scratch [9, 10] during fine-tuning. The work closest to ours is [7], where the authors condition a GPT-2 on prompts constructed using retrieved captions. However, the key difference between our work and theirs is that we require only a text-caption-only datastore for RECAP, whereas their system requires both audio and text pairs. We also introduce additional cross-attention layers for audio conditioning. Kim et. al [6], the current state-of-the-art system, proposed prefix tuning for audio captioning where the authors feed a prefix or a fixed-size embedding sequence to GPT-2 for audio captioning. Other works include synthetic data augmentation techniques [11, 12], and training tricks to improve learning on the source training data [13, 14].
自动音频描述。当前音频描述领域的研究主要采用编码器-解码器模型架构，其中描述文本由基于音频编码器所获表征的自回归语言解码器生成[5,6,7]。这些语言解码器要么基于网络规模数据预训练[5,6,7]，要么在微调阶段从头训练[9,10]。与本研究最接近的是文献[7]，作者通过检索获得的描述文本来构建 GPT-2 的提示条件。但核心区别在于我们的 RECAP 系统仅需纯文本描述数据库，而他们的系统需要音频-文本配对数据。我们还引入了额外的交叉注意力层来实现音频条件化。当前最先进的 Kim 等人[6]提出前缀调优方法，通过向 GPT-2 输入前缀或固定尺寸的嵌入序列来生成音频描述。其他研究包括合成数据增强技术[11,12]以及改进源训练数据学习的训练技巧[13,14]。

Retrieval-augmented Generation. The core idea of retrieval-augmented generation (RAG) is to condition generation on additional data retrieved from an external datastore [8]. RAG has been shown to benefit knowledge-intensive NLP tasks like open-domain question-answering on datasets that require world knowledge and advanced reasoning capabilities [15, 16]. RAG has also proven to be extremely effective in various computer vision tasks, including image captioning [17, 18]. We argue that audio captioning, especially in out-of-domain scenarios, is a knowledge-intensive task as it requires the model to caption novel audio concepts never seen during training, and can benefit from RAG.
检索增强生成。检索增强生成（RAG）的核心思想是基于从外部数据存储中检索到的附加数据来生成内容[8]。研究表明，RAG 能显著提升需要世界知识和高级推理能力的知识密集型 NLP 任务（如开放域问答）的性能[15,16]。该方法在包括图像描述生成在内的多种计算机视觉任务中也展现出卓越效果[17,18]。我们认为音频描述任务（尤其是域外场景）作为知识密集型任务，需要模型描述训练阶段从未接触过的新颖音频概念，因此同样能从 RAG 中获益。

Problem Formulation. Given a dataset $\mathcal{D}$ with audio-text pairs ($\mathcal{A}$,$\mathcal{T}$), where each text caption $t_{i}\in\mathcal{T}$ corresponding to an audio $a_{i}\in\mathcal{A}$ describes the content or events of the audio, we aim to train a model $\theta$ to generate $t_{i}$ from $a_{i}$. Different from other audio captioning systems, we also assume that the model has access to a datastore $\mathcal{DS}$ with text captions during inference. These captions come from the training set of $\mathcal{D}$ or external sources but have no overlap with the validation or test sets of $\mathcal{D}$.
问题定义。给定一个包含音频-文本对（ $\mathcal{A}$ ， $\mathcal{T}$ ）的数据集 $\mathcal{D}$ ，其中每个文本描述 $t_{i}\in\mathcal{T}$ 对应一段音频 $a_{i}\in\mathcal{A}$ ，用于描述该音频的内容或事件。我们的目标是训练一个模型 $\theta$ ，使其能够根据 $a_{i}$ 生成 $t_{i}$ 。与其他音频描述系统不同，我们假设该模型在推理过程中可以访问一个包含文本描述的数据存储 $\mathcal{DS}$ 。这些描述来自 $\mathcal{D}$ 的训练集或外部来源，但与 $\mathcal{D}$ 的验证集或测试集无重叠。

Datasets. For training and evaluating RECAP, we use either Clotho [3], AudioCaps [2], or a combination of both. Clotho has 3839/1045/1045 unique audios in train/dev/test splits, respectively, with five captions for each audio. AudioCaps has 49,838/495/975 with five captions except for the train set.
数据集。我们使用 Clotho[3]、AudioCaps[2]或两者组合来训练和评估 RECAP。Clotho 数据集在训练/开发/测试集中分别包含 3839/1045/1045 段独特音频，每段音频配有五条描述文本。AudioCaps 数据集包含 49,838/495/975 段音频，除训练集外每段音频也配有五条描述。

Baselines. We compare RECAP with six competitive baselines that are taken from literature. Eren et al. [9] and Xu et al. [10] train a Gated Recurrent Unit (GRU) for generating captions, conditioned on audio embeddings extracted from an audio encoder. Chen et al. [20] replaces the GRU with a transformer decoder, and Mei et al. [19] trains an entire encoder-decoder transformer architecture from scratch. Kim et al. [6] and Gontier et al. [5] use a pre-trained language model, where the former employs GPT-2, and the latter employs BART [24].
基线模型。我们将 RECAP 与文献中的六个竞争性基线模型进行对比。Eren 等人[9]和 Xu 等人[10]训练了门控循环单元(GRU)来生成描述文本，其条件输入为音频编码器提取的嵌入特征。Chen 等人[20]用 Transformer 解码器替代了 GRU，Mei 等人[19]则从头训练完整的编码器-解码器 Transformer 架构。Kim 等人[6]和 Gontier 等人[5]使用了预训练语言模型，前者采用 GPT-2，后者采用 BART[24]。

Experimental Setup. To compare the performance of RECAP, we conduct experiments in three distinct setups: (1) We train and evaluate the model on the same dataset $\mathcal{D}$, (2) We train the model on a dataset $\mathcal{D}$ and evaluate the model on a different dataset $\hat{\mathcal{D}}$ (3) We train the model on a combination of both datasets and evaluate separately on individual datasets. For (1), the datastore $\mathcal{DS}$ consists of captions from either the training set of the source dataset $\mathcal{D}$ or a large curated datastore $\mathcal{DS}_{large}$. For (2), we use $\mathcal{DS}$ that has captions from either $\mathcal{D}$ ($\mathcal{DS}$), $\mathcal{DS}_{large}$ or from the other dataset. For (3), we either use $\mathcal{DS}$ that has captions from both datasets or use $\mathcal{DS}_{large}$. We list all the sources of $\mathcal{DS}_{large}$ with over 600,000+ text-only captions on our GitHub. This includes 100,000+ new weakly labeled captions for the AudioSet strong subset and three new captions for each sample in AudioCaps and Clotho. All these captions were generated using GPT-4 and manually corrected by one expert human annotator. For retrieval-based prompt creation, we use $k$=4 and retrieve only the top 4 captions from the datastore. It is worth noting that RECAP does not use any additional training or data augmentation tricks. For both AudioCaps and Clotho, we train using Adam optimizer with a learning rate of 5e^-5 for 100 epochs and a batch size of 32. We evaluate all our models on the metrics of BLEU, METEOR, ROUGE-L, CIDEr, SPICE, and SPIDEr.
实验设置。为比较 RECAP 的性能，我们在三种不同配置下进行实验：(1)在同一数据集 $\mathcal{D}$ 上训练并评估模型，(2)在数据集 $\mathcal{D}$ 上训练模型并在不同数据集 $\hat{\mathcal{D}}$ 上评估，(3)在组合数据集上训练模型并分别在单个数据集上评估。对于(1)，数据存储 $\mathcal{DS}$ 包含来自源数据集 $\mathcal{D}$ 训练集的文本描述或大型精选数据存储 $\mathcal{DS}_{large}$ 。对于(2)，我们使用 $\mathcal{DS}$ ——其描述文本来自 $\mathcal{D}$ ( $\mathcal{DS}$ )、 $\mathcal{DS}_{large}$ 或另一数据集。对于(3)，我们使用包含双数据集描述的 $\mathcal{DS}$ 或采用 $\mathcal{DS}_{large}$ 。我们在 GitHub 上列出了所有 $\mathcal{DS}_{large}$ 来源，包含超过 60 万条纯文本描述，其中包括为 AudioSet 强子集新增的 10 万+弱标注描述，以及为 AudioCaps 和 Clotho 每个样本新增的三条描述。所有描述均使用 GPT-4 生成并经专业标注员人工校正。基于检索的提示创建中，我们设定 $k$ =4 且仅从数据存储中检索前 4 条最佳描述。 值得注意的是，RECAP 并未使用任何额外的训练技巧或数据增强手段。对于 AudioCaps 和 Clotho 数据集，我们均采用 Adam 优化器进行训练，学习率设置为 5e-4，训练 100 个 epoch，批次大小为 32。所有模型均通过 BLEU、METEOR、ROUGE-L、CIDEr、SPICE 和 SPIDEr 指标进行评估。

Results. Table 1 and Table 2 compare the performance of RECAP against all our baselines evaluated on Clotho and AudioCaps, respectively. We train our models in different settings and evaluate them with different datastores. While RECAP shows decent margins of improvement in in-domain settings, RECAP outperforms all baselines by a significant margin in out-of-domain settings when an in-domain datastore is available. Without one, RECAP shows competitive performance with SOTA [6]. The presence of a larger data store ($\mathcal{DS}_{large}$) almost always improves performance. This opens possibilities to improve captioning performance by augmenting the datastore with diverse synthetically generated captions.
实验结果。表 1 和表 2 分别展示了 RECAP 在 Clotho 和 AudioCaps 数据集上相较于所有基线模型的性能对比。我们在不同设置下训练模型，并使用不同数据存储进行评估。当存在领域内数据存储时，RECAP 在领域外设置中以显著优势超越所有基线模型，而在领域内设置中也展现出稳定的性能提升。若无领域内数据存储，RECAP 仍能与当前最佳模型 SOTA[6]保持相当性能。更大规模的数据存储（ $\mathcal{DS}_{large}$ ）几乎总能带来性能提升，这为通过合成多样化描述文本来扩充数据存储、从而提升字幕生成性能提供了可能性。

Results Analysis. Table 3 compares RECAP with Kimet al. [6] (SOTA) on compositional instances from Clotho (1.) and AudioCaps (4.) test set. While SOTA was able to caption only one audio event, due to conditioning on a prompt constructed from diverse retrieved captions, RECAP captures multiple. We also compared with a model trained on AudioCaps and inferred on a Clotho test instance with an audio event never seen during training (2.), and vice-versa (3.). By being conditioned on in-domain prompts, RECAP can caption these instances effectively.
结果分析。表 3 将 RECAP 与 Kimet al.[6]（当前最优模型）在 Clotho 测试集(1.)和 AudioCaps 测试集(4.)的组合实例上进行了对比。由于当前最优模型仅能描述单个音频事件，而 RECAP 通过基于多样化检索字幕构建的提示条件，实现了对多个事件的捕捉。我们还比较了在 AudioCaps 上训练后推断 Clotho 测试实例(2.)的模型（该音频事件在训练中从未出现），以及反向情况(3.)。通过基于领域内提示的条件设置，RECAP 能够有效描述这些实例。