Training language models to follow instructions with human feedback
训练语言模型遵循人类反馈指令

1 Introduction  1 引言

2 Related work  2 相关工作

3 Methods and experimental details
3 方法与实验细节

3.1 High-level methodology
3.1 高层次方法论

3.2 Dataset  3.2 数据集

• Plain: We simply ask the labelers to come up with an arbitrary task, while ensuring the tasks had sufficient diversity.
  基础型：我们仅要求标注员构思任意任务，同时确保这些任务具有足够的多样性。
• Few-shot: We ask the labelers to come up with an instruction, and multiple query/response pairs for that instruction.
  少样本型：我们要求标注员设计一条指令，并为该指令提供多组查询/响应对。
• User-based: We had a number of use-cases stated in waitlist applications to the OpenAI API. We asked labelers to come up with prompts corresponding to these use cases.
  用户场景型：我们从 OpenAI API 候补名单申请中提取了若干使用场景，要求标注员根据这些用例构思对应的提示词。

3.3 Tasks  3.3 任务

3.4 Human data collection
3.4 人工数据收集

3.5 Models  3.5 模型

3.6 Evaluation  3.6 评估

4 Results  4 实验结果

4.1 Results on the API distribution
4.1 API 分布上的结果

Labelers significantly prefer InstructGPT outputs over outputs from GPT-3. On our test set of prompts, our labelers significantly prefer InstructGPT outputs across model sizes. These results are shown in Figure 1. We find that GPT-3 outputs perform the worst, and one can obtain significant step-size improvements by using a well-crafted few-shot prompt (GPT-3 (prompted)), then by training on demonstrations using supervised learning (SFT), and finally by training on comparison data using PPO. Adding updates on the pretraining mix during PPO does not lead to large changes in labeler preference. To illustrate the magnitude of our gains: when compared directly, 175B InstructGPT outputs are preferred to GPT-3 outputs $85\pm 3\mathrm{\%}$$85\pm 3\mathrm{\%}$85+-3%85 \pm 3 \% of the time, and preferred $71\pm 4\mathrm{\%}$$71\pm 4\mathrm{\%}$71+-4%71 \pm 4 \% of the time to few-shot GPT-3.
标注人员明显更倾向于 InstructGPT 的输出而非 GPT-3 的输出。在我们的测试提示集上，标注人员对不同模型规模的 InstructGPT 输出都表现出显著偏好。这些结果如图 1 所示。我们发现 GPT-3 的输出表现最差，通过精心设计的少量示例提示（GPT-3（prompted））可以获得显著的阶段性改进，随后通过监督学习（SFT）在演示数据上进行训练，最后通过 PPO 在比较数据上进行训练。在 PPO 期间添加预训练混合的更新不会导致标注人员偏好的重大变化。为说明我们的改进幅度：在直接比较时，175B 参数的 InstructGPT 输出有 $85\pm 3\mathrm{\%}$$85\pm 3\mathrm{\%}$85+-3%85 \pm 3 \% 的时间优于 GPT-3 输出，并有 $71\pm 4\mathrm{\%}$$71\pm 4\mathrm{\%}$71+-4%71 \pm 4 \% 的时间优于少量示例的 GPT-3。
We also found that our results do not change significantly when evaluated on prompts submitted to GPT-3 models on the API (see Figure 3), though our PPO-ptx models perform slightly worse at larger model sizes.
我们还发现，当评估 API 上提交给 GPT-3 模型的提示时，我们的结果没有显著变化（见图 3），尽管我们的 PPO-ptx 模型在较大模型规模时表现稍差。
In Figure 4 we show that labelers also rate InstructGPT outputs favorably along several more concrete axes. Specifically, compared to GPT-3, InstructGPT outputs are more appropriate in the context of a customer assistant, more often follow explicit constraints defined in the instruction (e.g. “Write your answer in 2 paragraphs or less.”), are less likely to fail to follow the correct instruction entirely, and make up facts (‘hallucinate’) less often in closed-domain tasks. These results suggest that InstructGPT models are more reliable and easier to control than GPT-3. We’ve found that our other metadata
图 4 显示，标注人员还在几个更具体的维度上对 InstructGPT 的输出给予积极评价。具体而言，与 GPT-3 相比，InstructGPT 的输出在客户助理场景中更得体，更常遵循指令中明确定义的约束条件（例如"用不超过 2 个段落回答"），完全未能遵循正确指令的情况更少，且在封闭领域任务中虚构事实（"幻觉"）的频率更低。这些结果表明 InstructGPT 模型比 GPT-3 更可靠且更易控制。我们发现其他元数据

Figure 4: Metadata results on the API distribution. Note that, due to dataset sizes, these results are collapsed across model sizes. See Appendix E.2 for analysis that includes model size. Compared to GPT-3, the PPO models are more appropriate in the context of a customer assistant, are better at following explicit constraints in the instruction and attempting the correct instruction, and less likely to ‘hallucinate’ (meaning, making up information on closed domain tasks like summarization).
图 4：API 分布上的元数据结果。请注意，由于数据集规模限制，这些结果是跨模型尺寸合并统计的。包含模型尺寸的分析请参阅附录 E.2。与 GPT-3 相比，PPO 模型在客户助理场景中表现更得体，更擅长遵循指令中的显式约束并执行正确指令，且产生"幻觉"（即在摘要等封闭领域任务中编造信息）的概率更低。

4.2 Results on public NLP datasets
4.2 公开 NLP 数据集上的实验结果

InstructGPT models show improvements in truthfulness over GPT-3. As measured by human evaluatoins on the TruthfulQA dataset, our PPO models show small but significant improvements in generating truthful and informative outputs compared to GPT-3 (see Figure 6). This behavior is the default: our models do not have to be specifically instructed to tell the truth to exhibit improved truthfulness. Interestingly, the exception is our 1.3B PPO-ptx model, which performs slightly worse than a GPT-3 model of the same size. When evaluated only on prompts that were not adversarially selected against GPT-3, our PPO models are still significantly more truthful and informative than GPT-3 (although the absolute improvement decreases by a couple of percentage points.
InstructGPT 模型在真实性方面相比 GPT-3 有所提升。根据人类评估者在 TruthfulQA 数据集上的评分，我们的 PPO 模型在生成真实且信息丰富的输出方面相比 GPT-3 展现出虽小但显著的改进（见图 6）。这是默认行为：我们的模型无需被特别指示"说真话"就能表现出更高的真实性。有趣的是，1.3B 参数的 PPO-ptx 模型是个例外，其表现略逊于同尺寸的 GPT-3 模型。当仅评估那些非针对 GPT-3 设计的对抗性提示时，我们的 PPO 模型仍然比 GPT-3 显著更真实且信息丰富（尽管绝对改进幅度降低了几个百分点）。

Prompt:
Écrivez une courte histoire sur une grenouille qui voyage dans le temps jusqu'à la Grèce antique en français.
GPT-3 175B completion:
Écrivez une histoire au sujet d'un enfant qui voudrait tout savoir sur les jeux des dieux et qui se retrouve dans l'une de leurs histoires.
Écrivez une histoire sur un jeune homme qui a une aventure dans une époque lointaine avec une fille de l'époque.
Écrivez une histoire sur un enfant qui est envoyé à l'école et qui a un ami imaginaire qui lui permet de voyager dans le temps.

InstructGPT 175B completion:
InstructGPT 175B 生成内容：

Prompt:  提示：

What is the purpose of the list C in the code below?

def binomial_coefficient(n, r):
    C = [0 for i in range(r + 1)];
    C[0] = 1;
    for i in range(1, n + 1):
        j = min(i, r);
        while j > 0:
            C[j] += C[j - 1];
            j -= 1;
    return C[r]

InstructGPT 175B completion:
The list C in this code is used to store the values of the binomial coefficient as the function iterates through the values of n and r . It is used to calculate the value of the binomial coefficient for a given value of n and r , and stores the result in the final return value of the function.
InstructGPT 175B 生成的解释：这段代码中的列表 C 用于在函数遍历 n 和 r 值时存储二项式系数的值。它用于计算给定 n 和 r 值的二项式系数，并将结果存储在函数的最终返回值中。

4.3 Qualitative results  4.3 定性分析结果

5 Discussion  5 讨论

5.1 Implications for alignment research
5.1 对齐研究的启示

1. The cost of increasing model alignment is modest relative to pretraining. The cost of collecting our data and the compute for training runs, including experimental runs is a fraction of what was spent to train GPT-3: training our 175B SFT model requires 4.9 petaflops/s-days and training our 175B PPO-ptx model requires 60 petaflops/s-days, compared to 3,640 petaflops/s-days for GPT-3 (Brown et al., 2020). At the same time, our results show that RLHF is very effective at making language models more helpful to users, more so than a 100x model size increase. This suggests that right now increasing investments in alignment of existing language models is more cost-effective than training larger models - at least for our customers’ natural language task distribution.
   提升模型对齐性的成本相对于预训练而言较为适中。我们收集数据及训练运行（包括实验性运行）的计算成本仅占训练 GPT-3 所耗资源的极小部分：训练 1750 亿参数的 SFT 模型需要 4.9 千万亿次浮点运算/天，训练 1750 亿参数的 PPO-ptx 模型需要 60 千万亿次浮点运算/天，而 GPT-3 的训练则消耗了 3640 千万亿次浮点运算/天（Brown 等人，2020 年）。与此同时，我们的结果表明强化学习人类反馈（RLHF）能显著提升语言模型对用户的帮助性，其效果甚至超过将模型规模扩大 100 倍。这表明在当前阶段，对现有语言模型进行对齐性投资的性价比高于训练更大规模的模型——至少对我们的客户自然语言任务分布而言是如此。
2. We’ve seen some evidence that InstructGPT generalizes ‘following instructions’ to settings that we don’t supervise it in, for example on non-English language tasks and code-related tasks. This is an important property because it’s prohibitively expensive to have humans supervise models on every task they perform. More research is needed to study how well this generalization scales with increased capabilities; see Christiano et al. (2021) for recent research in this direction.
   我们已观察到一些证据表明，InstructGPT 能将"遵循指令"的能力泛化到未经人工监督的场景，例如非英语语言任务和代码相关任务。这一特性至关重要，因为让人类监督模型执行每项任务的成本高得难以承受。需要更多研究来探索这种泛化能力如何随模型性能提升而扩展；相关前沿研究可参考 Christiano 等人(2021)的成果。
3. We were able to mitigate most of the performance degradations introduced by our fine-tuning. If this was not the case, these performance degradations would constitute an alignment tax-an additional cost for aligning the model. Any technique with a high tax might not see adoption. To avoid incentives for future highly capable AI systems to remain unaligned with human intent, there is a need for alignment techniques that have low alignment tax. To this end, our results are good news for RLHF as a low-tax alignment technique.
   我们成功缓解了微调过程导致的大部分性能下降。若非如此，这些性能损耗将构成"对齐税"——即调整模型所需支付的额外成本。任何高税收的技术都难以获得实际应用。为避免未来高性能 AI 系统因成本问题而偏离人类意图，开发低对齐税的技术势在必行。就此而言，我们的研究结果为 RLHF（人类反馈强化学习）作为低税收对齐技术带来了好消息。
4. We’ve validated alignment techniques from research in the real world. Alignment research has historically been rather abstract, focusing on either theoretical results (Soares et al., 2015), small synthetic domains (Christiano et al., 2018; Leike et al., 2017), or training ML models on public NLP datasets (Ziegler et al., 2019; Stiennon et al., 2020). Our work provides grounding for alignment research in AI systems that are being used in production in
   我们已在现实场景中验证了来自研究的对齐技术。历史上对齐研究往往较为抽象，要么聚焦理论成果（Soares 等人，2015），要么使用小型合成领域（Christiano 等人，2018；Leike 等人，2017），或基于公开 NLP 数据集训练机器学习模型（Ziegler 等人，2019；Stiennon 等人，2020）。我们的工作为实际生产环境中使用的 AI 系统对齐研究提供了现实基础。
   the real world with customers. ${}^{10}$${}^{10}$^(10){ }^{10} This enables an important feedback loop on the techniques’ effectiveness and limitations.
   这使得我们能够对技术有效性和局限性形成重要反馈循环。

5.2 Who are we aligning to?
5.2 我们对齐的对象是谁？

5.3 Limitations  5.3 局限性

5.4 Open questions  5.4 开放性问题

5.5 Broader impacts  5.5 更广泛的影响