Agent AI:
Surveying the Horizons of Multimodal Interaction
智能体 AI：探索多模态交互的疆界

Agents that generate text are often prone to hallucinations, which are instances where the generated text is nonsensical or unfaithful to the provided source content Raunak et al. (2021); Maynez et al. (2020). Hallucinations can be split into two categories, intrinsic and extrinsic Ji et al. (2023). Intrinsic hallucinations are hallucinations that are contradictory to the source material, whereas extrinsic hallucinations are when the generated text contains additional information that was not originally included in the source material.
生成文本的智能体经常会出现幻觉现象，即生成的内容不符合逻辑或偏离原始素材（Raunak 等人，2021；Maynez 等人，2020）。幻觉可分为内在和外在两类（Ji 等人，2023）。内在幻觉指与源材料直接矛盾的内容，而外在幻觉则是生成文本包含源材料中未提及的额外信息。

Some promising routes for reducing the rate of hallucination in language generation involve using retrieval-augmented generation Lewis et al. (2020); Shuster et al. (2021) or other methods for grounding natural language outputs via external knowledge retrieval Dziri et al. (2021); Peng et al. (2023). Generally, these methods seek to augment language generation by retrieving additional source material and by providing mechanisms to check for contradictions between the generated response and the source material.
降低语言生成中幻觉率的有前景方法包括采用检索增强生成技术（Lewis 等人，2020；Shuster 等人，2021），或通过外部知识检索来锚定自然语言输出的其他方法（Dziri 等人，2021；Peng 等人，2023）。这些方法通常通过检索额外源材料，并提供机制检查生成响应与源材料间的矛盾，从而增强语言生成质量。

Within the context of multi-modal agent systems, VLMs have been shown to hallucinate as well Zhou et al. (2023b). One common cause of hallucination for vision-based language-generation is due to the over-reliance on co-occurrence of objects and visual cues in the training data Rohrbach et al. (2018). AI agents that exclusively rely upon pretrained LLMs or VLMs and use limited environment-specific finetuning can be particularly vulnerable to hallucinations since they rely upon the internal knowledge-base of the pretrained models for generating actions and may not accurately understand the dynamics of the world state in which they are deployed.
在多模态智能体系统中，视觉语言模型（VLM）同样存在幻觉现象（Zhou 等人，2023b）。基于视觉的语言生成出现幻觉的常见原因，是训练数据中物体共现和视觉线索的过度依赖（Rohrbach 等人，2018）。仅依赖预训练 LLMs 或 VLMs 且使用有限环境微调的 AI 智能体尤其容易产生幻觉，因为它们依赖预训练模型的内部知识库生成动作，可能无法准确理解所处环境的世界状态动态。

AI agents based on LLMs or LMMs (large multimodal models) have biases due to several factors inherent in their design and training process. When designing these AI agents, we must be mindful of being inclusive and aware of the needs of all end users and stakeholders. In the context of AI agents, inclusivity refers to the measures and principles employed to ensure that the agent’s responses and interactions are inclusive, respectful, and sensitive to a wide range of users from diverse backgrounds. We list key aspects of agent biases and inclusivity below.
基于 LLMs 或 LMMs（大型多模态模型）的 AI 智能体，其设计和训练过程中的固有因素会导致多种偏见。设计这些 AI 智能体时，我们必须注意包容性，关注所有终端用户和利益相关者的需求。在 AI 智能体语境下，包容性指为确保智能体响应和交互具有包容性、尊重性，并对不同背景用户保持敏感所采取的措施和原则。下文列出智能体偏见和包容性的关键方面。

• •

  Training Data: Foundation models are trained on vast amounts of text data collected from the internet, including books, articles, websites, and other text sources. This data often reflects the biases present in human society, and the model can inadvertently learn and reproduce these biases. This includes stereotypes, prejudices, and slanted viewpoints related to race, gender, ethnicity, religion, and other personal attributes. In particular, by training on internet data and often only English text, models implicitly learn the cultural norms of Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies Henrich et al. (2010) who have a disproportionately large internet presence. However, it is essential to recognize that datasets created by humans cannot be entirely devoid of bias, since they frequently mirror the societal biases and the predispositions of the individuals who generated and/or compiled the data initially.
  训练数据：基础模型通过从互联网收集的海量文本数据（包括书籍、文章、网站等）进行训练。这些数据往往反映人类社会存在的偏见，模型可能无意中学习并复现这些偏见，包括与种族、性别、民族、宗教等个人特征相关的刻板印象和片面观点。特别是通过互联网数据（通常仅为英语文本）训练时，模型会隐式学习西方发达工业国（WEIRD）社会的文化规范（Henrich 等人，2010），这些社会在互联网上占比过高。但需认识到人类创建的数据集无法完全消除偏见，因为它们常常反映社会偏见和原始数据生成/编译者的倾向。

• •

  Historical and Cultural Biases: AI models are trained on large datasets sourced from diverse content. Thus, the training data often includes historical texts or materials from various cultures. In particular, training data from historical sources may contain offensive or derogatory language representing a particular society’s cultural norms, attitudes, and prejudices. This can lead to the model perpetuating outdated stereotypes or not fully understanding contemporary cultural shifts and nuances.
  历史与文化偏见：AI 模型训练使用的海量数据集来源多样，常包含代表不同文化的历史文本资料。特别是历史源材料可能包含反映特定社会文化规范、态度和偏见的冒犯性语言，导致模型延续过时的刻板印象，或无法充分理解当代文化变迁与细微差异。

• •

  Language and Context Limitations: Language models might struggle with understanding and accurately representing nuances in language, such as sarcasm, humor, or cultural references. This can lead to misinterpretations or biased responses in certain contexts. Furthermore, there are many aspects of spoken language that are not captured by pure text data, leading to a potential disconnect between human understanding of language and how models understand language.
  语言与语境局限：语言模型可能难以理解和准确表达语言中的细微差别（如讽刺、幽默或文化典故），导致特定语境下的误解或偏见响应。此外，纯文本数据无法捕捉口语的诸多特征，造成人类语言理解与模型语言理解间的潜在脱节。

• •

  Policies and Guidelines: AI agents operate under strict policies and guidelines to ensure fairness and inclusivity. For instance, in generating images, there are rules to diversify depictions of people, avoiding stereotypes related to race, gender, and other attributes.
  政策与准则：人工智能代理需遵循严格的政策和准则，以确保公平性与包容性。例如在生成图像时，存在规范要求多样化呈现人物形象，避免涉及种族、性别等属性的刻板印象。

• •

  Overgeneralization: These models tend to generate responses based on patterns seen in the training data. This can lead to overgeneralizations, where the model might produce responses that seem to stereotype or make broad assumptions about certain groups.
  过度泛化：这些模型倾向于根据训练数据中的模式生成回应。这可能导致过度泛化，使模型可能产生看似刻板印象或对某些群体做出广泛假设的回应。

• •

  Constant Monitoring and Updating: AI systems are continuously monitored and updated to address any emerging biases or inclusivity issues. Feedback from users and ongoing research in AI ethics play a crucial role in this process.
  持续监控与更新：人工智能系统会持续受到监控和更新，以解决新出现的偏见或包容性问题。用户反馈和人工智能伦理领域的持续研究在此过程中发挥着关键作用。

• •

  Amplification of Dominant Views: Since the training data often includes more content from dominant cultures or groups, the model may be more biased towards these perspectives, potentially underrepresenting or misrepresenting minority viewpoints.
  主流观点放大效应：由于训练数据通常包含更多来自主流文化或群体的内容，模型可能更偏向这些视角，从而可能低估或错误呈现少数群体的观点。

• •

  Ethical and Inclusive Design: AI tools should be designed with ethical considerations and inclusivity as core principles. This includes respecting cultural differences, promoting diversity, and ensuring that the AI does not perpetuate harmful stereotypes.
  伦理与包容性设计：人工智能工具应以伦理考量和包容性为核心设计原则。这包括尊重文化差异、促进多样性，并确保人工智能不会延续有害的刻板印象。

• •

  User Guidelines: Users are also guided on how to interact with AI in a manner that promotes inclusivity and respect. This includes refraining from requests that could lead to biased or inappropriate outputs. Furthermore, it can help mitigate models learning harmful material from user interactions.
  用户指南：用户还会获得指导，了解如何以促进包容性和尊重的方式与人工智能互动。这包括避免提出可能导致偏见或不恰当输出的请求。此外，这也有助于防止模型从用户互动中学习有害内容。

Despite these measures, AI agents still exhibit biases. Ongoing efforts in agent AI research and development are focused on further reducing these biases and enhancing the inclusivity and fairness of agent AI systems. Efforts to Mitigate Biases:
尽管采取了这些措施，人工智能代理仍表现出偏见。当前代理人工智能的研发工作正致力于进一步减少这些偏见，提升代理人工智能系统的包容性和公平性。减轻偏见的措施包括：

• •

  Diverse and Inclusive Training Data: Efforts are made to include a more diverse and inclusive range of sources in the training data.
  多样化与包容性训练数据：致力于在训练数据中纳入更多样化和包容性的来源。

• •

  Bias Detection and Correction: Ongoing research focuses on detecting and correcting biases in model responses.
  偏见检测与修正：当前研究重点在于识别并修正模型响应中存在的偏见。

• •

  Ethical Guidelines and Policies: Models are often governed by ethical guidelines and policies designed to mitigate biases and ensure respectful and inclusive interactions.
  伦理准则与政策：模型通常遵循旨在减少偏见、确保尊重与包容性互动的伦理准则和政策框架。

• •

  Diverse Representation: Ensuring that the content generated or the responses provided by the AI agent represent a wide range of human experiences, cultures, ethnicities, and identities. This is particularly relevant in scenarios like image generation or narrative construction.
  多元表征：确保 AI 智能体生成的内容或提供的响应能体现人类经验、文化、族裔和身份的多样性。这在图像生成或叙事构建等场景中尤为重要。

• •

  Bias Mitigation: Actively working to reduce biases in the AI’s responses. This includes biases related to race, gender, age, disability, sexual orientation, and other personal characteristics. The goal is to provide fair and balanced responses that do not perpetuate stereotypes or prejudices.
  偏见缓解：积极减少 AI 响应中的偏见，包括与种族、性别、年龄、残障、性取向等个人特征相关的偏见，旨在提供不延续刻板印象的公平回应。

• •

  Cultural Sensitivity: The AI is designed to be culturally sensitive, acknowledging and respecting the diversity of cultural norms, practices, and values. This includes understanding and appropriately responding to cultural references and nuances.
  文化敏感性：AI 设计需具备文化敏感性，认知并尊重文化规范、习俗与价值观的多样性，包括准确理解文化典故与细微差异。

• •

  Accessibility: Ensuring that the AI agent is accessible to users with different abilities, including those with disabilities. This can involve incorporating features that make interactions easier for people with visual, auditory, motor, or cognitive impairments.
  可访问性：确保 AI 智能体适用于不同能力用户，包括残障人士，需整合辅助功能以方便视听觉障碍、行动或认知障碍者的交互。

• •

  Language-based Inclusivity: Providing support for multiple languages and dialects to cater to a global user base, and being sensitive to the nuances and variations within a language Liu et al. (2023b).
  语言包容性：支持多语言及方言以适应全球用户，敏感处理语言内部差异（Liu 等，2023b）。

• •

  Ethical and Respectful Interactions: The Agent is programmed to interact ethically and respectfully with all users, avoiding responses that could be deemed offensive, harmful, or disrespectful.
  道德与尊重交互：智能体编程需遵循道德规范，避免冒犯性、有害或不尊重的回应。

• •

  User Feedback and Adaptation: Incorporating user feedback to continually improve the inclusivity and effectiveness of the AI agent. This includes learning from interactions to better understand and serve a diverse user base.
  用户反馈与适应：通过用户反馈持续提升 AI 包容性，从交互中学习以更好服务多元用户群体。

• •

  Compliance with Inclusivity Guidelines: Adhering to established guidelines and standards for inclusivity in AI agent, which are often set by industry groups, ethical boards, or regulatory bodies.
  包容性准则合规：遵守行业组织、伦理委员会或监管机构制定的 AI 包容性标准。

Despite these efforts, it’s important to be aware of the potential for biases in responses and to interpret them with critical thinking. Continuous improvements in AI agent technology and ethical practices aim to reduce these biases over time. One of the overarching goals for inclusivity in agent AI is to create an agent that is respectful and accessible to all users, regardless of their background or identity.
尽管存在这些措施，仍需保持对潜在偏见的警觉并批判性解读响应内容。AI 技术的持续改进与伦理实践将逐步减少这些偏见。智能体 AI 包容性的核心目标是创建尊重所有用户且不受背景身份限制的服务。

One key ethical consideration of AI agents involves comprehending how these systems handle, store, and potentially retrieve user data. We discuss key aspects below:
AI 智能体的关键伦理考量涉及理解系统如何处理、存储及调用用户数据，主要方面如下：

The inference ability of an AI agent lies in its capacity to interpret, predict, and respond based on its training and input data. While these capabilities are advanced and continually improving, it’s important to recognize their limitations and the influence of the underlying data they are trained on. Particularly, in the context of large language models, it refers to its capacity to draw conclusions, make predictions, and generate responses based on the data it has been trained on and the input it receives. Inference augmentation in AI agents refers to enhancing the AI’s natural inference abilities with additional tools, techniques, or data to improve its performance, accuracy, and utility. This can be particularly important in complex decision-making scenarios or when dealing with nuanced or specialized content. We denote particularly important sources for inference augmentation below:
人工智能代理的推理能力体现在其基于训练数据和输入信息进行解释、预测与响应的能力。尽管这些能力十分先进且持续提升，但必须认识到其局限性及训练数据带来的影响。尤其在大型语言模型语境下，这种能力特指模型根据训练数据和接收的输入得出结论、进行预测并生成响应的能力。 AI 代理的推理增强是指通过附加工具、技术或数据来提升其自然推理能力，从而改善性能、准确性和实用性。这在复杂决策场景或处理微妙专业内容时尤为重要。我们特别标注以下对推理增强至关重要的来源：

Recently, Agent AI has made significant advancements, and its integration into embodied systems has opened new possibilities for interacting with agents via more immersive, dynamic, and engaging experiences. To expedite the process and ease the cumbersome work in agent AI developing, we are proposing to develop the next-generation AI-empowered pipeline for agent interaction. Develop a human-machine collaboration system where humans and machines can communicate and interact meaningfully. The system can leverage the LLM’s or VLM dialog capabilities and vast action to talk with human players and identify human needs. Then it will perform proper actions to help human players upon request.
近期，智能体人工智能取得重大进展，其与具身系统的融合为通过更沉浸、动态和吸引人的体验与智能体交互开辟了新可能。为加速进程并减轻智能体 AI 开发中的繁重工作，我们提议开发新一代支持 AI 的智能体交互管道。构建人机协作系统，使人类与机器能够进行有意义的沟通与互动。该系统可利用 LLM 或 VLM 的对话能力与海量行为库，与人类玩家交谈并识别需求，随后根据请求执行恰当动作来协助玩家。

When employing LLM/VLMs for a human-machine collaboration system, it is essential to note that these operate as black boxes, generating unpredictable output. This uncertainty can become crucial in a physical setup, such as operating actual robotics. An approach to address this challenge is constraining the focus of the LLM/VLM through prompt engineering. For instance, in robotic task planning from instructions, providing environmental information within the prompt has been reported to yield more stable outputs than relying solely on text Gramopadhye and Szafir (2022). This report is supported by the Minsky’s frame theory of AI Minsky (1975), suggesting that the problem space to be solved by LLM/VLMs is defined by the given prompts. Another approach is designing prompts to make LLM/VLMs include explanatory text to allow users understand what the model has focused on or recognized. Additionally, implementing a higher layer that allows for pre-execution verification and modification under human guidance can facilitate the operation of systems working under such guidance (Fig. 4).
在将 LLM/VLM 应用于人机协作系统时，必须注意它们作为黑箱运行会产生不可预测的输出。这种不确定性在物理场景（如操作真实机器人）中尤为关键。应对这一挑战的方法是通过提示工程来约束 LLM/VLM 的焦点。例如在基于指令的机器人任务规划中，研究表明在提示中包含环境信息比仅依赖文本能产生更稳定的输出（Gramopadhye 和 Szafir，2022）。该结论与 Minsky（1975）提出的 AI 框架理论相呼应，表明 LLM/VLM 待解决的问题空间由给定提示定义。另一策略是设计提示使 LLM/VLM 生成解释性文本，让用户理解模型的关注点或识别内容。此外，构建允许人工指导下进行执行前验证与修改的高层控制机制，可提升此类指导型系统的可操作性（图 4）。

There is a rich history of leveraging reinforcement learning (RL) to train interactive agents that exhibits intelligent behaviors. RL is a methodology to learn the optimal relationship between states and actions based on rewards (or penalties) received as a result of its actions. RL is a highly scalable framework that has been applied to numerous applications including robotics, however, it generally faces several leader-board and LLM/VLMs have shown their potential to mitigate or overcome some of those difficulties:
利用强化学习（RL）训练具有智能行为的交互式智能体已有丰富研究历史。RL 是通过行为导致的奖励（或惩罚）来学习状态与动作间最优关系的方法论。作为高度可扩展的框架，RL 已应用于机器人等众多领域，但通常面临若干挑战。当前领先的 LLM/VLM 模型已展现出缓解或克服部分困难的潜力：

• •

  Reward designing The efficiency of policy learning greatly depends on the design of the reward function. Designing the reward function requires not only knowledge of RL algorithms but also a deep understanding of the nature of the task, and thus often necessitates crafting the function based on expert experience. Several studies explored the use of LLM/VLMs for designing reward functions Yu et al. (2023a); Katara et al. (2023); Ma et al. (2023).
  奖励设计 策略学习效率极大依赖于奖励函数的设计。设计奖励函数不仅需要 RL 算法知识，还需深刻理解任务本质，因此往往需基于专家经验构建。多项研究探索了使用 LLM/VLM 设计奖励函数的方法（Yu 等人，2023a；Katara 等人，2023；Ma 等人，2023）。

• •

  Data collection and efficiency Given its exploratory nature, RL-based policy learning requires a significant amount of data Padalkar et al. (2023). The necessity for extensive data becomes particularly evident when the policy involves managing long sequences or integrating complex actions. This is because these scenarios demand more nuanced decision-making and learning from a wider range of situations. In recent studies, efforts have been directed towards enhancing data generation to support policy learning Kumar et al. (2023); Du et al. (2023). Additionally, in some studies, these models have been integrated into the reward function to improve policy learning Sontakke et al. (2023). Parallel to these developments, another strand of research has focused on achieving parameter efficiency in learning processes using VLMs Tang et al. (2023); Li et al. (2023d) and LLMs Shi et al. (2023)
  数据收集与效率 基于强化学习的策略学习因其探索性质需要大量数据（Padalkar 等人，2023）。当策略涉及管理长序列或整合复杂动作时，对海量数据的需求尤为明显，因为这些场景需要更精细的决策能力，并要求从更广泛的情境中学习。近期研究致力于通过增强数据生成来支持策略学习（Kumar 等人，2023；Du 等人，2023）。部分研究还将这些模型整合至奖励函数中以优化策略学习（Sontakke 等人，2023）。与此同时，另一研究方向聚焦于利用视觉语言模型（Tang 等人，2023；Li 等人，2023d）和 LLMs（Shi 等人，2023）实现学习过程的参数高效化。

• •

  Long-horizon steps In relation to the issue of data efficiency, RL becomes more challenging as the length of action sequences increases. This is due to the ambiguity in the relationship between actions and rewards, known as the credit assignment problem, and the increase in the number of states to be explored, necessitating a significant amount of time and data. One typical approach for long and complex tasks is to break them down into a sequence of subgoals and apply pretrained policies to solve each subgoal (e.g., Takamatsu et al. (2022)). This idea falls within the framework called the task and motion planning (TAMP)Garrett et al. (2021). TAMP is composed of two primary components: task planning, which entails identifying sequences of high-level actions, and motion planning, which involves finding physically consistent, collision-free trajectories to achieve the objectives of the task plan.
  长时程步骤 关于数据效率问题，随着动作序列长度的增加，强化学习面临更大挑战。这源于动作与奖励关联的模糊性（即信用分配问题），以及需探索状态数量的增长，需要耗费大量时间和数据。针对长周期复杂任务的典型解决方案是将其分解为一系列子目标，并应用预训练策略逐个解决（如 Takamatsu 等人 2022 年研究）。这一思路属于任务与运动规划（TAMP）框架范畴（Garrett 等人 2021 年）。TAMP 由两大核心组件构成：任务规划（确定高层动作序列）和运动规划（寻找物理合理、无碰撞的轨迹以实现任务规划目标）。

  LLMs are well-suited to TAMP, and recent research has often adopted an approach where LLMs are used to execute high-level task planning, while low-level controls are addressed with RL-based policies Xu et al. (2023); Sun et al. (2023a); Li et al. (2023b); Parakh et al. (2023). The advanced capabilities of LLMs enable them to effectively decompose even abstract instructions into subgoals Wake et al. (2023c), contributing to the enhancement of language understanding abilities in robotic systems.
  LLMs 天然适配 TAMP 框架，近期研究多采用 LLMs 执行高层任务规划，同时用基于强化学习的策略处理底层控制（Xu 等人 2023；Sun 等人 2023a；Li 等人 2023b；Parakh 等人 2023）。LLMs 的先进能力使其能有效将抽象指令分解为子目标（Wake 等人 2023c），从而提升机器人系统的语言理解能力。

While RL aims to train a policy based on exploratory behavior and maximizing rewards through interactions with the environment, imitation learning (IL) seeks to leverage expert data to mimic the actions of experienced agents or experts. For example, in robotics, one of the major frameworks based on IL is Behavioral Cloning (BC). BC is an approach where a robot is trained to mimic the actions of an expert by directly copying them. In this approach, the expert’s actions in performing specific tasks are recorded, and the robot is trained to replicate these actions in similar situations. Recent BC-based methods often incorporate technologies from LLM/VLMs, enabling more advanced end-to-end models. For example, Brohan et al. proposed RT-1 Brohan et al. (2022) and RT-2 Brohan et al. (2023), transformer-based models that output an action sequence for the base and arm, taking a series of images and language as input. These models are reported to show high generalization performance as the result of training on a large amount of training data.
强化学习（RL）旨在通过与环境交互的探索行为来训练策略并最大化奖励，而模仿学习（IL）则试图利用专家数据来模仿经验丰富的智能体或专家的行为。例如在机器人领域，基于 IL 的主要框架之一是行为克隆（BC）。该方法通过直接复制专家动作来训练机器人进行模仿：专家执行特定任务时的动作被记录下来，机器人被训练在类似情境中复现这些动作。近期基于 BC 的方法常整合 LLM/VLMs 技术，实现了更先进的端到端模型。例如 Brohan 等人提出的 RT-1（2022）和 RT-2（2023）是基于 Transformer 的模型，以系列图像和语言作为输入，输出底座和机械臂的动作序列。据报道，这些模型通过海量训练数据展现出优异的泛化性能。

Learning intelligent agent behavior leveraging image inputs has been of interest for many years Mnih et al. (2015). The inherent challenge of using RGB input is the curse of dimensionality. To solve this problem, researchers either use more data Jang et al. (2022); Ha et al. (2023) or introduce inductive biases into the model design to improve sample efficiency. In particular, authors incorporate 3D structures into the model architecture for manipulations Zeng et al. (2021); Shridhar et al. (2023); Goyal et al. (2023); James and Davison (2022). For robot navigation, authors Chaplot et al. (2020a, b) leverage maps as a representation. Maps can either be learned from a neural network aggregating all previous RGB inputs or through 3D reconstruction methods such as Neural Radiance Fields Rosinol et al. (2022).
利用图像输入学习智能体行为多年来备受关注（Mnih 等人，2015）。使用 RGB 输入的本质挑战在于维度灾难。为解决该问题，研究者要么使用更多数据（Jang 等人，2022；Ha 等人，2023），要么在模型设计中引入归纳偏置以提高样本效率。特别是有些作者将 3D 结构整合到模型架构中以处理操作任务（Zeng 等人，2021；Shridhar 等人，2023；Goyal 等人，2023；James 和 Davison，2022）。在机器人导航领域，Chaplot 等人（2020a，b）利用地图作为表征方式——地图既可通过神经网络聚合所有历史 RGB 输入来学习，也能通过 Neural Radiance Fields 等 3D 重建方法获得（Rosinol 等人，2022）。

To obtain more data, researchers synthesize synthetic data using graphics simulators Mu et al. (2021); Gong et al. (2023b), and try to close the sim2real gap Tobin et al. (2017); Sadeghi and Levine (2016); Peng et al. (2018). Recently, there has been some collective effort to curate large-scale dataset that aims to resolve the data scarcity problem Padalkar et al. (2023); Brohan et al. (2023). On the other hand, to improve sample complexity, data augmentation techniques have been extensively studied as well Zeng et al. (2021); Rao et al. (2020); Haarnoja et al. (2023); Lifshitz et al. (2023).
为获取更多数据，研究者使用图形模拟器合成人工数据（Mu 等人，2021；Gong 等人，2023b），并试图缩小仿真与现实间的差距（Tobin 等人，2017；Sadeghi 和 Levine，2016；Peng 等人，2018）。近期学界开始协力构建旨在解决数据稀缺问题的大规模数据集（Padalkar 等人，2023；Brohan 等人，2023）。另一方面，为提高样本复杂度，数据增强技术也得到广泛研究（Zeng 等人，2021；Rao 等人，2020；Haarnoja 等人，2023；Lifshitz 等人，2023）。

In-context learning was shown to be an effective method for solving tasks in NLP with the advent of large language models like GPT-3 Brown et al. (2020); Min et al. (2022). Few-shot prompts were seen to be an effective way to contextualize model output’s across a variety of tasks in NLP by providing examples of the task within the context of the LLM prompt. Factors like the diversity of examples and quality of examples shown for the in-context demonstrations may improve the quality of model outputs An et al. (2023); Dong et al. (2022). Within the context of multi-modal foundation models, models like Flamingo and BLIP-2 Alayrac et al. (2022); Li et al. (2023c) have been shown to be effective at a variety of visual understanding tasks when given only given a small number of examples. In context learning can be further improved for agents within environments by incorporating environment-specific feedback when certain actions are taken Gong et al. (2023a).
随着 GPT-3 等大型语言模型的出现，上下文学习已被证明是 NLP 任务的有效解决方法（Brown 等，2020；Min 等，2022）。少量示例提示能通过 LLM 上下文中提供的任务样本，使模型输出适配各类 NLP 任务。研究表明，上下文演示中示例的多样性与质量等因素可提升模型输出质量（An 等，2023；Dong 等，2022）。在多模态基础模型领域，Flamingo 和 BLIP-2 等模型（Alayrac 等，2022；Li 等，2023c）仅需少量示例即可高效完成多种视觉理解任务。通过引入环境特异性行为反馈（Gong 等，2023a），可进一步优化环境中智能体的上下文学习能力。

The optimization of agent systems can be divided into spatial and temporal aspects. Spatial optimization considers how agents operate within a physical space to execute tasks. This includes inter-robot coordination, resource allocation, and keeping an organized space.
智能体系统优化可分为空间与时间两个维度。空间优化关注智能体在物理环境中执行任务的方式，包括机器人间协同、资源分配及空间组织管理。

In order to effectively optimize agent AI systems, especially systems with large numbers of agents acting in parallel, previous works have focused on using large batch reinforcement learning Shacklett et al. (2023). Since datasets of multi-agent interactions for specific tasks are rare, self-play reinforcement learning enables a team of agents to improve over time. However, this may also lead to very brittle agents that can only work under self-play and not with humans or other independent agents since they over-fit to the self-play training paradigm. To address this issue, we can instead discover a diverse set of conventions Cui et al. (2023); Sarkar et al. (2023), and train an agent that is aware of a wide range of conventions. Foundation models can further help to establish conventions with humans or other independent agents, enabling smooth coordination with new agents.
为有效优化 Agent AI 系统（尤其是多智能体并行系统），前人研究主要采用大批量强化学习（Shacklett 等，2023）。由于特定任务的多智能体交互数据集稀缺，自我对弈强化学习能使智能体团队持续进化，但可能导致智能体过度适应训练范式而无法与人类或其他独立智能体协作。针对此问题，可采用多样化策略发现（Cui 等，2023；Sarkar 等，2023）来训练具备广谱策略认知的智能体。基础模型能进一步帮助建立人机或智能体间协作规范，实现与新智能体的无缝协调。

Temporal optimization, on the other hand, focuses on how agents execute tasks over time. This encompasses task scheduling, sequencing, and timeline efficiency. For instance, optimizing the trajectory of a robot’s arm is an example of efficiently optimizing movement between consecutive tasks Zhou et al. (2023c). At the level of task scheduling, methods like LLM-DP Dagan et al. (2023) and ReAct Yao et al. (2023a) have been proposed to solve efficient task planning by incorporating environmental factors interactively.
时间优化则侧重于智能体如何随时间推移执行任务，这包括任务调度、序列安排和时间线效率。例如，优化机械臂的运动轨迹就是高效协调连续任务间动作的典型案例（Zhou 等人，2023c）。在任务调度层面，LLM-DP（Dagan 等人，2023）和 ReAct（Yao 等人，2023a）等方法通过动态融合环境因素，提出了高效的交互式任务规划解决方案。

Our foray into the agent paradigm involves the development of Agent AI "Modules" for interactive multi-modal agents using LLMs or VLMs. Our initial Agent Modules facilitate training or in-context learning and adopt a minimalist design for the purposes of demonstrating the agent’s ability to schedule and coordinate effectively. We also explored initial prompt-based memory techniques that facilitate better planning and inform future actions approaches within the domain. To illustrate, our “MindAgent" infrastructure comprises 5 main modules: 1) environment perception with task planning, 2) agent learning, 3) memory, 4) general agent action prediction and 5) cognition, as shown in Figure 5.
我们对智能体范式的探索涉及开发基于 LLMs 或 VLMs 的多模态交互式 Agent AI"模块"。初始的智能体模块采用极简设计，支持训练或上下文学习，旨在展示智能体有效调度与协调的能力。我们还研究了基于提示的初级记忆技术，这些技术能促进领域内更优的规划并为未来行动方法提供依据。以"MindAgent"架构为例，其包含五大核心模块：1) 带任务规划的环境感知，2) 智能体学习，3) 记忆，4) 通用智能体行为预测，以及 5) 认知系统，如图 5 所示。

Agent-based AI is a large and fast-growing community within the domains of entertainment, research, and industry. The development of large foundation models has significantly improved the performance of agent AI systems. However, creating agents in this vein is limited by the increasing effort necessary to create high-quality datasets and overall cost. At Microsoft, building high-quality agent infrastructure has significantly impacted multi-modal agent copilots by using advanced hardware, diverse data sources, and powerful software libraries. As Microsoft continues to push the boundaries of agent technology, AI agent platforms are poised to remain a dominant force in the world of multimodal intelligence for years to come. Nevertheless, agent AI interaction is currently still a complex process that requires a combination of multiple skills. The recent advancements in the space of large generative AI models have the potential to greatly reduce the current high cost and time required for interactive content, both for large studios, as well as empowering smaller independent content creators to design high quality experiences beyond what they are currently capable of. The current human-machine interaction systems inside multi-modal agents are primarily rule-based. They do have intelligent behaviors in response to human/user actions and possess web knowledge to some extent. However, these interactions are often limited by software development costs to enable specific behaviors in the system. In addition, current models are not designed to help human to achieve a goal in the case of users’ inability to achieve specific tasks. Therefore, there is a need for an agent AI system infrastructure to analyze users behaviors and provide proper support when needed.
基于智能体的 AI 在娱乐、研究和工业领域已形成庞大且快速发展的生态。大型基础模型的发展显著提升了 Agent AI 系统的性能，但这类智能体的创建正面临高质量数据集构建成本攀升的限制。微软通过先进硬件、多元数据源和强大软件库构建的高质量智能体基础设施，对多模态智能体协作者系统产生了深远影响。随着微软持续突破智能体技术边界，AI 智能体平台有望在未来多年保持多模态智能领域的主导地位。然而当前 Agent AI 交互仍是需要多技能结合的复杂过程——大型生成式 AI 模型的最新进展，既能帮助大型工作室大幅降低交互内容的高成本与耗时，也能赋能独立创作者突破现有能力边界设计高质量体验。现有多模态智能体中的人机交互系统主要基于规则设计，虽能响应用户行为并具备一定网络知识，但其交互能力常受限于实现特定系统行为的开发成本。更重要的是，当用户无法完成特定任务时，现有模型缺乏辅助达成目标的机制，因此亟需能分析用户行为并在必要时提供支持的 Agent AI 系统基础设施。

Action agents refer to the agents that need to execute physical actions in the simulated physical environment or real world. In particular, they need to be actively engaging in activities with the environment. We broadly classify action agents into two different categories based on their application domains: gaming AI and robotics.
动作智能体是指在模拟物理环境或现实世界中需要执行物理动作的智能体。这类智能体需主动与环境进行互动。我们根据应用领域将动作智能体大致分为两类：游戏人工智能和机器人技术。

In gaming AI, the agents will interact with the game environment and other independent entities. In these settings, natural language can enable smooth communication between agents and humans. Depending on the game, there may be a specific task to accomplish, providing a true reward signal. For instance, in the competitive Diplomacy game, training a language model using human conversation data along with an action policy with RL enables human-level play Meta Fundamental AI Research Diplomacy Team et al.(2022)Meta Fundamental AI Research (FAIR) Diplomacy Team, Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, et al. (FAIR).
在游戏 AI 中，智能体将与游戏环境及其他独立实体互动。这类场景中，自然语言能实现智能体与人类的流畅沟通。根据游戏类型，可能需要完成特定任务以获取真实奖励信号。例如在竞技游戏《外交》中，通过结合人类对话数据训练语言模型，并采用强化学习策略，可实现人类水平的游戏表现（Meta 基础 AI 研究外交团队等，2022）。

There are also settings where we agents act as normal residents in a town Park et al. (2023a), without trying to optimize a specific goal. Foundation models are useful in these settings because they can model interactions that appear more natural by mimicking human behavior. When augmented with external memory, they produce convincing agents that can have conversations, daily schedules, form relationships, and have a virtual life.
在某些场景中，我们让智能体作为普通居民生活在虚拟城镇里（Park 等人，2023a），而无需优化特定目标。基础模型在此类场景中极具价值，因为它们能通过模拟人类行为来呈现更自然的交互。当辅以外置记忆功能时，这些模型能生成具备对话能力、日常作息、社交关系乃至完整虚拟人生的可信智能体。

Interactive agents simply refer to agents that can interact with the world, a broader class of agents than action agents. Their forms of interaction do not necessarily require physical actions, but may involve communicating information to users or modifying the environment. For instance, an embodied interactive agent may answer a user’s questions about a topic through dialogue or help users parse through existing information similar to a chatbot. By extending an agent’s capabilities to include information sharing, the core designs and algorithms of Agent AI can be effectively adapted for a range of applications, such as diagnostic Lee et al. (2023) and knowledge-retrieval Peng et al. (2023) agents.
交互式智能体简指能与世界互动的智能体，这是比行动智能体更广泛的类别。它们的互动形式不一定需要物理动作，可能包括向用户传递信息或修改环境。例如，一个具身交互智能体可以通过对话回答用户关于某个主题的疑问，或像聊天机器人那样帮助用户解析现有信息。通过扩展智能体的信息共享能力，Agent AI 的核心设计与算法可有效适配多种应用场景，如诊断类（Lee 等人，2023）和知识检索类（Peng 等人，2023）智能体。

AR/VR/mixed-reality (jointly referred to as XR) settings currently require skilled artists and animators to create characters, environments, and objects to be used to model interactions in virtual worlds. This is a costly process that involves concept art, 3D modeling, texturing, rigging, and animation. XR agents can assist in this process by facilitating interactions between creators and building tools to help build the final virtual environment.
当前 AR/VR/混合现实（统称 XR）领域需要专业美术师和动画师创建角色、场景与交互物体，这一过程涉及概念设计、3D 建模、贴图、骨骼绑定和动画制作等高成本环节。XR 智能体可通过促进创作者间协作、开发虚拟环境构建工具来优化这一流程。

Our early experiments have already demonstrated that GPT models can be used in the few-shot regime inside of the Unity engine (without any additional fine-tuning) to call engine-specific methods, use API calls to download 3d models from the internet and place them into the scene, and assign state trees of behavior and animations to them Huang et al. (2023a). This behavior likely emerges due to the presence of similar code in open source game repositories that use Unity. Therefore, GPT models are capable of building rich visual scenes in terms of loading in many objects into the scene from a simple user prompt.
我们早期的实验已经证明，GPT 模型可以在 Unity 引擎中以少量示例模式（无需任何额外微调）调用引擎特定方法，使用 API 调用从互联网下载 3D 模型并放置到场景中，还能为其分配行为状态树和动画（Huang 等人，2023a）。这种行为可能源于开源游戏仓库中存在使用 Unity 的类似代码。因此，GPT 模型能够根据简单的用户提示，通过向场景加载大量对象来构建丰富的视觉场景。

The aim of this category of agents is to build a platform and a set of tools that provide an efficient interface between large AI models (both GPT-family ones as well as diffusion image models) and a rendering engine. We explore two primary avenues here:
这类智能体的目标是构建一个平台和工具集，为大型 AI 模型（包括 GPT 系列模型和扩散图像模型）与渲染引擎之间提供高效接口。我们在此探索两条主要路径：

• •

  Integration of large models into the various editor tools in the agent infrastructure, allowing for significant speedups in development.
  将大型模型集成到智能体基础设施的各类编辑工具中，从而显著提升开发效率。

• •

  Controlling the rendering engine from within a user experience, by generating code that follows user instruction and then compiling it at runtime, allowing for users to potentially edit the VR/simulation they are interacting with in arbitrary ways, even by introducing new agent mechanics.
  通过在用户体验中生成遵循用户指令的代码并实时编译，实现对渲染引擎的操控。这使得用户能以任意方式（甚至通过引入新的智能体机制）编辑他们正在交互的 VR/模拟环境。

Introducing an AI copilot focused on XR settings would be useful for XR creators, who can use the copilot to complete tedious tasks, like providing simple assets or writing code boilerplate, freeing creators to focus on their creative vision and quickly iterate on ideas.
为 XR 创作者引入专注于扩展现实场景的 AI 协作者将大有裨益。创作者可利用该协作者完成繁琐任务（如提供简单素材或编写代码模板），从而解放精力专注于创意构想并快速迭代想法。

Furthermore, agents can help users interactively modify the environment by adding new assets, changing the dynamics of the environment, or building new settings. This form of dynamic generation during runtime can also be specified by a creator, enabling the user’s experience to feel fresh and continue evolving over time.
此外，智能体还能帮助用户通过添加新素材、改变环境动态或构建新场景来交互式修改环境。这种运行时动态生成模式也可由创作者预先设定，使用户体验保持新鲜感并持续进化。

Knowledge Agents reason over their acquired knowledge systems in two directions: implicit and explicit. Implicit knowledge is typically what large-scale language models like the GPT series Brown et al. (2020); OpenAI (2023) encapsulate after being trained on vast amounts of text data. These models can generate responses that give the impression of understanding, as they draw on patterns and information implicitly learned during training. Explicit knowledge, conversely, is structured and can be directly queried, such as the information found in knowledge bases or databases, which was traditionally used to enhance AI reasoning capabilities by referencing verifiable external resources.
知识智能体通过隐性和显性两种方式对其掌握的知识体系进行推理。隐性知识通常是像 GPT 系列这样的大规模语言模型（Brown 等人，2020；OpenAI，2023）通过海量文本数据训练后所内化的内容。这些模型能生成看似理解语义的回应，因为它们利用了训练过程中隐性习得的模式与信息。相反，显性知识具有结构化特征且可直接查询，例如知识库或数据库中的信息——传统上正是通过引用可验证的外部资源来增强 AI 推理能力。

Despite the advancements in language models, their implicit knowledge is static and becomes outdated as the world evolves Lewis et al. (2020); Peng et al. (2023). This limitation necessitates the integration of explicit knowledge sources that are updated continuously, ensuring that AI systems can provide accurate and current responses. The fusion of implicit and explicit knowledge equips AI agents with a more nuanced understanding and the ability to apply knowledge contextually, akin to human intelligence Gao et al. (2022). Such integration is crucial for crafting knowledge-centric AI agents that not only possess information but can also understand, explain, and employ it, thereby narrowing the chasm between extensive learning and profound knowledge (Marcus and Davis, 2019; Gao et al., 2020). These agents are designed to reason with flexibility and dynamic information about the world, enhancing their robustness and adaptability (Marcus, 2020).
尽管语言模型取得了长足进步，但其隐性知识具有静态性，会随着世界发展而过时（Lewis 等人，2020；Peng 等人，2023）。这种局限性要求整合持续更新的显性知识源，以确保 AI 系统能提供准确且与时俱进的响应。隐性知识与显性知识的融合赋予 AI 智能体更细腻的认知能力，使其能像人类智能那样进行情境化知识应用（Gao 等人，2022）。这种集成对于构建以知识为中心的 AI 智能体至关重要——它们不仅掌握信息，更能理解、解释并运用知识，从而弥合广博学习与深刻认知之间的鸿沟（Marcus 和 Davis，2019；Gao 等人，2020）。这类智能体被设计成能灵活推理动态世界信息，从而增强其鲁棒性与适应性（Marcus，2020）。

Generally, a logic agent is a component of a system designed to apply logical reasoning to process data or solve tasks specific to logical inference or logical reasoning. Logic agents within the context of large foundation models like GPT-4 refers to a specialized component or submodules designed to handle logical reasoning tasks. These tasks often involve understanding and manipulating abstract concepts, deducing conclusions from given premises, or solving problems that require a structured, logical approach. Broadly, foundation models like GPT-4 are trained on a vast corpus of text data and learn to perform a wide range of tasks, including those that require some form of logical reasoning. Thus, their capability for logical reasoning is integrated into the overall architecture, and they generally do not possess a distinct, isolated "Logic agent". While GPT-4 and similar models can perform tasks that involve logic, their approach is fundamentally different from how humans or traditional logic-based systems operate. They do not follow formal logical rules or have an explicit understanding of logic; rather, they generate responses based on patterns learned from the training data. As a result, their performance in logical tasks can be impressive, but it can also be inconsistent or limited by the nature of the training data and the inherent limitations of the model’s design. One example of embedding a separate logical submodule into the architecture is (Wang et al., 2023e), which modifies the token embedding process used by LLMs during pre-training by parsing text into logical segments and explicitly modeling logical hierarchies in the token embeddings.
通常而言，逻辑智能体是系统的一个组件，旨在运用逻辑推理来处理数据或解决与逻辑推断、逻辑推理相关的特定任务。在 GPT-4 等大型基础模型的语境下，逻辑智能体指的是专门处理逻辑推理任务的组件或子模块。这类任务通常涉及理解与操作抽象概念、从给定前提推导结论，或解决需要结构化逻辑方法的问题。广义上说，GPT-4 等基础模型通过海量文本数据训练获得多样化任务处理能力，包括需要某种形式逻辑推理的任务。因此，其逻辑推理能力已融入整体架构，通常不存在独立分离的"逻辑智能体"。虽然 GPT-4 及类似模型能执行涉及逻辑的任务，但其运作方式与人类或传统逻辑系统存在本质差异。这些模型不遵循形式逻辑规则，也不具备对逻辑的显式理解，而是基于训练数据中的模式生成响应。因此，它们在逻辑任务中的表现可能令人惊艳，但也可能因训练数据特性和模型设计固有的局限性而出现不一致或受限的情况。Wang 等人(2023e)的研究展示了在架构中嵌入独立逻辑子模块的实例：通过将文本解析为逻辑片段并在词嵌入中显式建模逻辑层级，改进了 LLMs 预训练阶段的词嵌入过程。

Emotional understanding and empathy are important skills for agents in many human-machine interactions. To illustrate, one important goal for creating engaging dialogue agents is to have the agents act with increased emotion and empathy while minimizing socially inappropriate or offensive outputs. To advance towards this goal for dialogue agents, we released the Neural Image Commenting with Empathy (NICE) dataset Chen et al. (2021) consisting of almost two million images and the corresponding human-generated comments and a set of human emotion annotations. We also provided a novel pre-training model - Modeling Affect Gneration for Image Comments (MAGIC) Chen et al. (2021) - which aims to generate comments for images, conditioned on linguistic representations that capture style and affect, and to help generate more empathetic, emotional, engaging and socially appropriate comments. Our experiments show that the approach is effective in training a more human-like and engaging image comment agent. Developing empathy-aware agents is a promising direction for interactive agents, and it is important to create agents with emotional understanding capabilities across a wide range of groups and populations, especially considering that many current language models exhibit bias in their emotional understanding and empathetic reasoning capabilities Mao et al. (2022); Wake et al. (2023d).
情感理解与共情能力是许多人机交互场景中智能体的重要技能。以对话智能体为例，其核心目标之一是在最大限度减少社交失当或冒犯性输出的同时，增强情感表达与共情能力。为此，我们发布了神经共情图像评论数据集 NICE（Chen 等，2021），包含近两百万张图片及对应的人类评论与情感标注。同时提出了创新预训练模型 MAGIC（Chen 等，2021），该模型基于捕捉风格与情感的语言表征生成图像评论，有助于产出更具同理心、情感丰富且社交得体的内容。实验证明该方法能有效训练出更拟人化、更具吸引力的图像评论智能体。发展具备共情意识的智能体是交互代理的重要方向，尤其需注意当前许多语言模型在情感理解与共情推理方面存在偏见（Mao 等，2022；Wake 等，2023d），因此创建具有跨群体情感理解能力的代理至关重要。

Neuro-Symbolic agents operate on a hybrid system of neurons and symbols d’Avila Garcez and Lamb (2020). To solve problems stated in natural language is a challenging task because it requires explicitly capturing discrete symbolic structural information implicit in the input. However, most general neural sequence models do not explicitly capture such structural information, limiting their performance on these tasks. The work Chen et al. (2020) propose a new encoder-decoder model based on a structured neural representation agent, The encoder of TP-N2F employs TPR ‘binding’ to encode natural-language symbolic structure in vector space and the decoder uses TPR ‘unbinding’ to generate, in symbolic space, a sequential program represented by relational tuples, each consisting of a relation (or operation) and a number of arguments.
神经符号智能体基于神经元与符号的混合系统运作(d’Avila Garcez 与 Lamb，2020)。解决自然语言表述的问题具有挑战性，因为这需要显式捕捉输入中隐含的离散符号结构信息。然而，大多数通用神经序列模型并未显式捕获此类结构信息，限制了其在这些任务上的表现。Chen 等人(2020)提出了一种基于结构化神经表征智能体的新型编码器-解码器模型。TP-N2F 的编码器采用 TPR"绑定"技术在向量空间中编码自然语言的符号结构，解码器则通过 TPR"解绑"在符号空间生成由关系元组构成的序列程序，每个元组包含一个关系(或操作)及若干参数。

Instruction following vision-language (VL) models like GPT-4 offer a flexible interface that supports a broad range of multimodal tasks in a zero-shot fashion. However, interfaces that operate on full images do not directly enable the user to “point to” and access specific regions within images. This capability is important not only to support reference-grounded VL benchmarks, but also, for practical applications that require precise within-image reasoning. In (Park et al., 2023b), we build Localized Visual Commonsense model which allows users to specify (multiple) regions-as-input. We train our model by sampling localized commonsense knowledge from a large language model (LLM): specifically, we prompt a LLM to collect common sense knowledge given a global literal image description and a local literal region description automatically generated by a set of VL models. This pipeline is scalable and fully automatic, as no aligned or human-authored image and text pairs are required. With a separately trained critic model that selects high quality examples, we find that training on the localized commonsense corpus expanded solely from images can successfully distill existing VL models to support a reference-as-input interface. Empirical results and human evaluations in zero-shot settings demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression.
诸如 GPT-4 等遵循指令的视觉语言(VL)模型提供了灵活接口，支持以零样本方式处理多种多模态任务。但基于完整图像操作的接口无法让用户直接"指向"并访问图像中的特定区域。这种能力不仅对支持基于参照的 VL 基准测试至关重要，也对需要精确图像内推理的实际应用具有重要意义。Park 等人(2023b)构建的局部视觉常识模型允许用户指定(多个)输入区域。我们通过从大型语言模型(LLM)中采样局部常识知识进行训练：具体而言，我们提示 LLM 根据 VL 模型自动生成的全局字面图像描述和局部字面区域描述来收集常识知识。该流程具有可扩展性和全自动性，无需人工对齐或标注的图文配对数据。借助单独训练的选择高质量样本的评判模型，我们发现仅通过图像扩展的局部常识语料库训练，能有效蒸馏现有 VL 模型以支持参照输入接口。零样本设置下的实证结果与人工评估表明，相比传递生成参照表达的基线方法，我们的蒸馏方法能产生更精确的 VL 推理模型。

In modern gaming systems, the behavior of Non-Player Characters (NPCs) is predominantly dictated by predefined scripts crafted by developers. These scripts encompass a range of reactions and interactions based on various triggers or player actions within the gaming environment. However, this scripted nature often results in predictable or repetitive NPC behavior which fails to evolve in response to player’s actions or the dynamic environment of the game. This rigidity hampers the immersive experience intended in a dynamic gaming environment. Therefore, there is a burgeoning interest in leveraging LLMs to induce autonomy and adaptability in NPC behavior, making interactions more nuanced and engaging. AI-driven NPCs can learn from player behavior, adapt to varying strategies, and provide a more challenging and less predictable gameplay experience. Large Language Models (LLMs) can significantly contribute to evolving NPC behavior in games. By processing vast amounts of text, LLMs can learn patterns and generate responses that are more varied and human-like. They can be utilized to create dynamic dialogue systems, making interactions with NPCs more engaging and less predictable. Furthermore, LLMs can be trained on player feedback and in-game data to continually refine NPC behaviors, making them more attuned to player expectations and game dynamics.
在现代游戏系统中，非玩家角色（NPC）的行为主要受开发者预设脚本的支配。这些脚本包含基于游戏环境中各种触发条件或玩家行为的一系列反应与互动机制。然而，这种脚本化的本质往往导致 NPC 行为呈现可预测性或重复性，无法根据玩家行为或游戏动态环境进行演进。这种僵化特性阻碍了动态游戏环境本该具有的沉浸式体验。因此，业界日益关注利用 LLMs 赋予 NPC 行为自主性与适应性，使互动更具层次感和吸引力。由 AI 驱动的 NPC 能够学习玩家行为模式，适应不同策略，从而提供更具挑战性且难以预测的游戏体验。大型语言模型（LLMs）对游戏 NPC 行为的演进具有重要价值：通过处理海量文本数据，LLMs 能学习行为模式并生成更丰富、更拟人化的反馈；可用于构建动态对话系统，使 NPC 互动更具吸引力且难以预料；此外，LLMs 还能基于玩家反馈和游戏内数据进行持续训练，不断优化 NPC 行为，使其更符合玩家预期和游戏动态。

The interaction between human players and NPCs is a crucial aspect of the gaming experience. The conventional interaction paradigm is primarily one-dimensional, with NPCs reacting in a preset manner to player inputs. This limitation stifles the potential for a more organic and enriching interaction, akin to human-human interaction within the virtual realm. The advent of LLM and VLM technologies holds the promise of transforming this paradigm. By employing these technologies, gaming systems can analyze and learn from human behavior to provide more human-like interactions. This not only enhances the realism and engagement of the game but also provides a platform for exploring and understanding human-machine interaction in a controlled yet complex setting.
玩家与 NPC 之间的互动是游戏体验的关键要素。传统互动模式本质上是一维的，NPC 仅以预设方式对玩家输入做出反应。这种局限性抑制了虚拟世界中更自然、更丰富的互动潜力——这种潜力本应接近人与人之间的交互水平。LLM 和 VLM 技术的出现为改变这一范式带来了希望。通过运用这些技术，游戏系统可以分析并学习人类行为，从而提供更具人性化的互动。这不仅提升了游戏的真实感和沉浸感，也为在受控而复杂的环境中探索和理解人机交互提供了平台。

Gaming is an integral part of daily life, estimated to engage half of the world’s population³³3https://www.dfcint.com/global-video-game-audience-reaches-3-7-billion/. Additionally, it exhibits a positive impact on mental health⁴⁴4https://news.microsoft.com/source/features/work-life/mind-games-how-gaming-can-play-a-positive-role-in-mental-health/. However, contemporary game systems exhibit a deficiency in interactions with human players since their behaviors are primarily hand-crafted by game developers. These pre-programmed behaviors frequently fail to adapt to players’ needs. Consequently, there exists a need for new AI systems in games that can analyze player behaviors and furnish appropriate support when necessary. Intelligent interactive systems bear the potential to revolutionize how gamers interact with gaming systems in general. NPCs’ interactions with gamers are no longer confined by the restricted rule sets designed by game developers. They have the potential to adapt seamlessly to gamers’ experiences, providing timely feedback to enrich the gaming experience and elevate the synergy of human-machine interaction.
电子游戏已成为日常生活的重要组成部分，据估计全球半数人口参与其中 ³³3https://www.dfcint.com/global-video-game-audience-reaches-3-7-billion/ 。此外，游戏对心理健康具有积极影响 ⁴⁴4https://news.microsoft.com/source/features/work-life/mind-games-how-gaming-can-play-a-positive-role-in-mental-health/ 。然而，当代游戏系统存在与人类玩家互动的缺陷，因为其行为主要由游戏开发者手工设计。这些预设行为往往难以适应玩家需求。因此，游戏领域需要新型 AI 系统来分析玩家行为，并在必要时提供适当支持。智能交互系统有望彻底改变玩家与游戏系统的互动方式——非玩家角色（NPC）与玩家的互动不再受限于开发者设计的固定规则集，而是能够无缝适应玩家体验，通过实时反馈来丰富游戏乐趣并提升人机协同水平。

LLMs can serve as a robust tool for analyzing in-game text data, including chat logs, player feedback, and narrative content. They can help in identifying patterns of player behavior, preferences, and interactions which can be invaluable for game developers to improve game mechanics and narratives. Additionally, VLMs can parse through large quantities of image and video data from gaming sessions to help analyze user intent and actions within the game world. Moreover, LLMs and VLMs can facilitate the development of intelligent agents within games that can communicate with players and other agents in a sophisticated and human-like manner, enhancing the overall gaming experience. Beyond LLMs and VLMs, user input data, provides a promising avenue for creating game-playing agents that model perception, game playing, and game understanding by imitating human players. By incorporating a combination of player interactions and feedback, pixel inputs, and natural language planning and understanding, agent models can assist in the continuous improvement of game dynamics, driving a more player-centric evolution of the gaming environment.
LLMs 可作为分析游戏内文本数据（包括聊天记录、玩家反馈和叙事内容）的强大工具，帮助识别玩家行为模式、偏好与互动特征，这些对开发者改进游戏机制和叙事极具价值。视觉语言模型（VLMs）则可解析游戏过程中产生的大量图像和视频数据，辅助分析玩家在游戏世界中的意图与行为。此外，LLMs 与 VLMs 能促进游戏内智能体开发，使其能以复杂拟人的方式与玩家及其他智能体交流，从而提升整体游戏体验。除这两种模型外，用户输入数据为创建游戏代理提供了新途径——通过模仿人类玩家来建模感知、游戏操作与游戏理解。结合玩家互动反馈、像素输入以及自然语言规划理解，智能体模型能持续优化游戏动态，推动游戏环境向以玩家为中心的方向演进。

Scene synthesis is a vital component in the creation and enhancement of immersive gaming environments. It entails the automatic or semi-automatic generation of three-dimensional (3D) scenes and environments within a game. This process includes the generation of terrain, placement of objects, creation of realistic lighting, and sometimes even dynamic weather systems.
场景合成是创建和增强沉浸式游戏环境的核心环节，涉及游戏中三维场景与环境的全自动/半自动生成，包括地形生成、物体布局、真实光照创建，有时甚至包含动态天气系统构建。

Modern games often feature vast, open-world environments. Manually designing these landscapes can be incredibly time-consuming and resource-intensive. Automated terrain generation, often leveraging procedural or AI-driven techniques, can produce complex, realistic landscapes with less manual effort. LLMs and VLMs can utilize the internet scale knowledge to formulate rules to design non-repeating landscapes that are visually impressive and unique. Additionally, LLMs and VLMs can be used to ensure the semantic consistency and variability of generated assets. Placing objects such as buildings, vegetation, and other elements within a scene in a realistic and aesthetically pleasing manner is crucial for immersion.
现代游戏常以广袤的开放世界为特色，而人工设计这些场景耗时耗力。采用程序化或 AI 驱动技术的自动化地形生成，能以较少人工投入创造出复杂逼真的地貌。LLMs 与 VLMs 可运用互联网规模知识制定规则，设计出视觉震撼且永不重复的地形景观。同时，这些模型还能确保生成资源的语义一致性与多样性。在场景中以真实美观的方式布置建筑、植被等元素，对维持沉浸感至关重要。

VLMs and LLMs can assist in object placement by adhering to predefined or learned rules and aesthetics, thus speeding up the level design process. VLMs and LLMs can be further trained to understand the principles of design and aesthetics, aiding in the procedural generation of content. They can help formulate rules or guidelines that procedural algorithms can follow to generate objects, and scenes that are both visually appealing and contextually appropriate.
视觉语言模型(VLMs)和大型语言模型(LLMs)能够通过遵循预设或习得的规则与美学标准来辅助物体摆放，从而加速关卡设计流程。这些模型经过进一步训练后可以理解设计与美学原理，辅助内容的程序化生成。它们能帮助制定规则或指导方针，供程序化算法遵循以生成既具有视觉吸引力又符合情境需求的对象和场景。

Realistic lighting and atmospheric effects are fundamental for creating a believable and engaging gaming environment. Advanced algorithms can simulate natural lighting conditions and dynamic weather effects, enhancing the realism and mood of the scene. LLMs can help develop systems to acheive more realistic lighting and atmospheric effects in several innovative ways. VLMs can analyze vast datasets from real-world lighting and atmospheric conditions to help develop more realistic algorithms for simulating these effects in games. By understanding the patterns and intricacies of natural lighting and weather, these models can contribute to the development of algorithms that mimic reality closely. LLMs and VLMs could also be used to develop systems that adjust lighting and atmospheric effects in real-time based on player actions, game states, or external inputs. They can process natural language commands from players to modify the game environment, providing a more interactive and immersive experience.
逼真的光照与大气效果是构建可信且引人入胜游戏环境的基础。先进算法能模拟自然光照条件与动态天气效果，增强场景的真实感与氛围营造。LLMs 可通过多种创新方式协助开发实现更真实光照与大气效果的系统。VLMs 能分析来自现实世界光照与大气条件的海量数据集，协助开发更真实的游戏效果模拟算法。通过理解自然光照与天气的规律和复杂性，这些模型有助于开发能精准模拟现实的算法。LLMs 和 VLMs 还可用于开发基于玩家操作、游戏状态或外部输入实时调整光照与大气效果的系统，并能处理玩家的自然语言指令来修改游戏环境，提供更具交互性和沉浸感的体验。

Recent research has demonstrated the potential of LLM/VLMs for robotic agents that involve interactions with humans in an environment. Research topics that aim to leverage latest LLM/VLM technologies include:
最新研究表明，LLM/VLMs 在需与环境及人类交互的机器人智能体领域具有潜力。旨在利用前沿 LLM/VLM 技术的研究方向包括：

Multimodal Systems: Recent research has been actively focusing on developing end-to-end systems that incorporate the latest LLM and VLM technologies as encoders for input information. Particularly, there is a significant trend towards modifying these foundation models to process multimodal information. Jiang et al. (2022); Brohan et al. (2023, 2022); Li et al. (2023d); Ahn et al. (2022b); Shah et al. (2023b); Li et al. (2023e). This adaptation aims to guide robotic actions based on both linguistic instructions and visual cues, thus achieving an effective embodiment.
多模态系统：当前研究正积极开发以内置最新 LLM 和 VLM 技术作为输入信息编码器的端到端系统。尤其值得注意的是改造这些基础模型以处理多模态信息的显著趋势（Jiang 等人 2022；Brohan 等人 2023,2022；Li 等人 2023d；Ahn 等人 2022b；Shah 等人 2023b；Li 等人 2023e）。这种改造旨在通过语言指令和视觉线索共同引导机器人行为，从而实现有效具身化。

Task Planning and Skill Training: In contrast to end-to-end systems, Task And Motion Planning (TAMP) based systems first compute a high-level task plan and then achieve them with low-level robot control, known as skills.
任务规划与技能训练：与端到端系统不同，基于任务与运动规划（TAMP）的系统先计算高层任务计划，再通过被称为技能的底层机器人控制实现目标。

The advanced language processing abilities of LLMs have demonstrated the capability to interpret instructions and decompose them into robot action steps, greatly advancing task planning technologies Ni et al. (2023); Li et al. (2023b); Parakh et al. (2023); Wake et al. (2023c). For skill training, several studies have explored the use of LLMs/VLMs for designing reward functions Yu et al. (2023a); Katara et al. (2023); Ma et al. (2023), generating data to facilitate policy learning Kumar et al. (2023); Du et al. (2023), or serving as part of a reward function Sontakke et al. (2023). Together with training frameworks such as RL and IL, these efforts will contribute to the development of efficient robot controllers.
LLMs 先进的语言处理能力已展现出解析指令并将其分解为机器人行动步骤的潜力，极大推进了任务规划技术的发展（Ni 等人 2023；Li 等人 2023b；Parakh 等人 2023；Wake 等人 2023c）。在技能训练方面，多项研究探索了使用 LLMs/VLMs 设计奖励函数（Yu 等人 2023a；Katara 等人 2023；Ma 等人 2023）、生成促进策略学习的数据（Kumar 等人 2023；Du 等人 2023）或作为奖励函数组成部分（Sontakke 等人 2023）。这些努力与强化学习（RL）、模仿学习（IL）等训练框架相结合，将推动高效机器人控制器的发展。

On-site Optimization: Executing long task steps in robotics can be difficult due to unexpected and unpredictable environmental conditions. Therefore, a significant challenge in the field of robotics involves dynamically adapting and refining robotic skills by integrating task plans with real-time environmental data. For instance, (Ahn et al., 2022b) proposed an approach that calculates the feasibility of actions (i.e., affordance) from visual information and compares it with planned tasks. Additionally, there are approaches that focus on enabling LLMs to output the pre-conditions and post-conditions (e.g., states of objects and their interrelationships) of task steps to optimize their execution Zhou et al. (2023c) and detect pre-condition errors for necessary revisions to the task plan Raman et al. (2023). These strategies seek to achieve environment-grounded robot execution by integrating environmental information and adjusting the robot’s actions at the task plan or controller level.
现场优化：由于环境中存在意外且不可预测的条件，在机器人技术中执行长时任务步骤可能具有挑战性。因此，该领域的一大难题在于如何通过将任务规划与实时环境数据相结合，动态调整并完善机器人技能。例如，(Ahn 等人，2022b)提出了一种方法，通过视觉信息计算动作的可行性（即可供性）并与规划任务进行对比。此外，还有研究致力于让 LLMs 输出任务步骤的前置条件与后置条件（如物体状态及其相互关系）以优化执行过程（Zhou 等人，2023c），以及检测前置条件错误从而对任务规划进行必要修正（Raman 等人，2023）。这些策略旨在通过整合环境信息并在任务规划或控制器层面调整机器人动作，实现基于环境感知的机器人执行。

Conversation Agents: In creating conversational robots, LLMs can contribute to natural, context-sensitive interactions with humans Ye et al. (2023a); Wake et al. (2023f). These models process and generate responses that mimic human conversation, allowing robots to participate in meaningful dialogues. Additionally, LLMs play a significant role in the estimation of conceptual Hensel et al. (2023); Teshima et al. (2022) and emotional attributes Zhao et al. (2023); Yang et al. (2023b); Wake et al. (2023d) of utterances. Those attributes facilitate the understanding of human intent and meaningful gesture generation, thus contributing to the naturalness and efficacy of human-robot communication.
会话代理：在开发对话机器人时，LLMs 能够助力实现与人类自然、情境敏感的交互(Ye 等, 2023a; Wake 等, 2023f)。这些模型通过处理并生成拟人化回应，使机器人能够参与有意义的对话。此外，LLMs 在话语概念属性(Hensel 等, 2023; Teshima 等, 2022)和情感特征(Zhao 等, 2023; Yang 等, 2023b; Wake 等, 2023d)的评估中发挥关键作用。这些特性有助于理解人类意图并生成符合语境的动作，从而提升人机交互的自然度与有效性。

Navigation Agents: Robot navigation has a long history of research, focusing on core aspects such as map-based path planning and Simultaneous Localization and Mapping (SLAM) for creating environmental maps. These functionalities have become standard in widely used robot middleware like the Robot Operating System (ROS) Guimarães et al. (2016).
导航智能体：机器人导航研究历史悠久，其核心聚焦于基于地图的路径规划以及用于构建环境地图的同步定位与建图（SLAM）技术。这些功能已在机器人操作系统（ROS）等广泛使用的机器人中间件中成为标准配置（Guimarães 等人，2016）。

While classic navigation techniques remain prevalent in many robotics applications, they typically rely on static or pre-created maps. Recently, there has been an increased interest in advanced technologies that enable robots to navigate in more challenging environments, leveraging breakthroughs in fields like computer vision and natural language processing. One representative task is object navigation Chaplot et al. (2020a); Batra et al. (2020); Gervet et al. (2023); Ramakrishnan et al. (2022); Zhang et al. (2021), where robots use object names for navigation instead of map coordinates, requiring the visual grounding of object names in the environment. Furthermore, recent attention has been given to technologies that navigate robots in entirely unfamiliar new environments on a zero-shot basis, on top of foundation models, so-called zero-shot object navigation Gadre et al. (2023); Dorbala et al. (2023); Cai et al. (2023). Additionally, Vision-Language Navigation (VLN) Anderson et al. (2018a) is a representative task, where the task involves navigating an agent by natural language instructions in previously unseen, real-world environments Shah et al. (2023a); Zhou et al. (2023a); Dorbala et al. (2022); Liang et al. (2023); Huang et al. (2023b). VLN interprets sentences rather than object names, such as “go to the bathroom on your left.,” thus it requires a higher functionality to parse input text Wang et al. (2019).
虽然经典导航技术仍在多数机器人应用中占主导地位，但它们通常依赖静态或预构建地图。近年来，随着计算机视觉和自然语言处理领域的突破，针对复杂环境导航的先进技术受到越来越多的关注。代表性任务包括以物体名称而非地图坐标进行导航的物体导航(Chaplot 等, 2020a; Batra 等, 2020; Gervet 等, 2023; Ramakrishnan 等, 2022; Zhang 等, 2021)，这需要将物体名称与环境视觉信息进行锚定。更前沿的是基于基础模型的零样本物体导航技术(Gadre 等, 2023; Dorbala 等, 2023; Cai 等, 2023)，使机器人能在完全陌生的新环境中实现零样本导航。视觉语言导航(VLN)(Anderson 等, 2018a)则是另一项代表性任务，要求智能体根据自然语言指令（如"向左转进入浴室"）在未知的真实环境中导航(Shah 等, 2023a; Zhou 等, 2023a; Dorbala 等, 2022; Liang 等, 2023; Huang 等, 2023b)，这需要更高阶的文本解析能力(Wang 等, 2019)。

The advent of foundation models contributes to the development of such adaptive, on-the-fly navigation technologies by enhancing the understanding of human language instructions and the visual interpretation of environmental information. More detailed explanations of representative VLN research are provided in 6.2.2.
基础模型的出现推动了自适应实时导航技术的发展，通过增强对人类语言指令的理解和环境视觉信息的解读能力。关于视觉语言导航(VLN)代表性研究的详细阐述见 6.2.2 章节。

An accumulating body of evidence suggests that recent VLMs and LLMs have promising capabilities for symbolic task planning (e.g., what-to-do). However, each task requires low-level control policy (e.g., how-to-do) to achieve successful interaction between the environment. While reinforcement learning and imitation learning are promising approach to learn policies in a data-driven manner, another promising approach is to obtain the strategy directly from humans through on-site demonstration, an approach called Learning-from-Observation Wake et al. (2021a); Ikeuchi et al. . In this section, we introduce a study where we employ ChatGPT for task planning and enrich the plan by parameterizing it with affordance information to facilitate effective and precise execution (Fig. 13).
现有研究表明，最新视觉语言模型(VLMs)和 LLMs 在符号化任务规划(如"做什么")方面展现潜力。然而每个任务都需要底层控制策略(如"如何做")来实现与环境的有效交互。虽然强化学习和模仿学习是以数据驱动方式获取策略的有效途径，另一种可行方案是通过现场演示直接从人类获取操作策略——即"观察学习"方法(Wake 等, 2021a; Ikeuchi 等)。本节介绍我们利用 ChatGPT 进行任务规划，并通过功能可供性信息参数化来增强计划可执行性的研究(见图 13)。

The pipeline was composed of two modules: task planning and parameterization. In task planning, the system is fed with language instructions and the description of the working environment. These instructions, along with a predefined set of robot actions and output specifications, are compiled into a comprehensive prompt provided to ChatGPT, which then generates a sequence of decomposed tasks with their textual descriptions (Fig. 13; left pane). Notably, we employ a few-shot approach, meaning ChatGPT is not trained on this task, offering an advantage in applicability as it eliminates the need for hardware-dependent data collection and model training. Additionally, the textual descriptions in the output enable the user to check and adjust the results as necessary, which is a crucial feature for a safe and robust operation. Fig. 14 shows the qualitative results conducted for an agentic simulation on top of VirtualHome Puig et al. (2018). The results demonstrate a reasonable task plan and its flexibility in adjusting outputs, indicating the broad applicability of our approach.
该系统流程包含任务规划与参数化两大模块。任务规划模块接收语言指令和工作环境描述，结合预定义的机器人动作集与输出规范，编译成综合提示词输入 ChatGPT，后者生成带文本描述的任务分解序列(图 13 左窗格)。值得注意的是，我们采用小样本学习方式，ChatGPT 无需针对该任务进行训练，这种硬件无关的特性显著提升了适用性。输出中的文本描述允许用户核查和调整结果，这对确保系统安全稳健运行至关重要。图 14 展示了在 VirtualHome 平台(Puig 等, 2018)上进行的智能体仿真定性结果，验证了任务规划的合理性及其输出调整的灵活性，表明该方法具有广泛适用性。

While the task planner guarantees coherency between the task sequences, successful operation in reality requires detailed parameters. For example, grasp type is crucial for carrying a container while spilling out the content, such a parameter is often ignored in a simulators (see Fig. 14 in grasping a pie). In our robot system, therefore, users are asked to demonstrate each action visually (Fig. 13; right pane). The tasks had predefined parameters necessary for execution, which our vision system extracts from the videos Wake et al. (2021b). Notably, our robotic system is not designed for exact replication of human motions (i.e., teleoperation) but rather to handle varying real-world conditions, such as changes in object locations. Hence, the parameters extracted from human demonstrations encompass not precise motion paths but affordance information that dictates effective environmental movement (e.g., waypoints for collision avoidance Wake et al. (2023a), grasp types Wake et al. (2023e), and upper-limbs postures Sasabuchi et al. (2021); Wake et al. (2021a)). The posture of the upper limbs is critical in robots with high degrees of freedom and is designed to assume predictable postures for humans coexisting with the operational robot. The task sequence endowed with affordances is transformed into a sequence of reusable robot skills acquired through reinforcement learning and executed by the robot Takamatsu et al. (2022).
虽然任务规划器能确保任务序列间的连贯性，但在现实中的成功操作需要详细参数。例如，抓取类型对于搬运容器同时避免内容物洒出至关重要，这类参数在仿真环境中常被忽略（见图 14 抓取馅饼的示例）。因此，在我们的机器人系统中，用户需通过视觉演示每个动作（图 13 右侧面板）。任务包含执行所需的预定义参数，这些参数由我们的视觉系统从视频中提取（Wake 等人，2021b）。值得注意的是，我们的机器人系统并非设计用于精确复制人类动作（即遥操作），而是为了应对现实世界中多变的条件，例如物体位置变化。因此，从人类示范中提取的参数并非精确的运动路径，而是决定有效环境移动的功用信息（如避碰航点（Wake 等人，2023a）、抓取类型（Wake 等人，2023e）和上肢姿势（Sasabuchi 等人，2021；Wake 等人，2021a））。上肢姿势对于高自由度机器人至关重要，其设计需确保与操作机器人共处的人类能够预判其姿态。赋予功用信息的任务序列会被转化为通过强化学习获得的可复用机器人技能序列，并由机器人执行（Takamatsu 等人，2022）。

LLM-empowered task planning can be extended to a more versatile robotic system by integrating it with VLMs. Here, we show an example where we use the GPT-4V(ision) to broaden the aforementioned task planner in a multimodal input context (Fig. 15), a human performs actions that are intended to be replicated by the robot. In this paper, only part of the prompt is shown. The whole prompt is available at microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts.
通过将 LLM 驱动的任务规划与视觉语言模型（VLM）结合，可将其扩展至更通用的机器人系统。如图 15 所示，我们在多模态输入场景中使用 GPT-4V(ision)拓展了前述任务规划器——人类演示的动作将被机器人复现。本文仅展示部分提示词，完整提示详见 microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts。

This pipeline takes demonstration videos and text, then outputs a sequence of robot actions. A vision analyzer aims to understand the actions performed by humans in the video. We used GPT-4V and provided a prompt to generate text instructions in a style typical of human-to-human communication.Fig. 16 demonstrates how the usage of text input allows user to give feedback on GPT-4V’s recognition results for correction purposes. Such a feature, aiming at improving the accuracy of the recognition results, also enables more robust operation.
该流程接收演示视频和文本输入，随后输出机器人动作序列。视觉分析模块旨在理解视频中人类的操作行为。我们采用 GPT-4V 模型，通过提示指令生成符合人际交流风格的文本说明。图 16 展示了如何通过文本输入让用户对 GPT-4V 的识别结果提供修正反馈。这种旨在提升识别准确度的功能，同时也能实现更稳定的系统运作。

Next, the scene analyzer compiles the expected work environment into the text information based on the instructions and the first frame of the video data (or an image of the environment). This environmental information includes a list of object names recognized by GPT-4V, the graspable properties of objects, and the spatial relationships between objects. Although these computational processes are a black box within GPT-4V, the information is output based on the knowledge of GPT-4V and the image/text input. Fig. 17 shows the example outputs of our scene analyzer. As shown in the figure, GPT-4V successfully selects the objects that are related to the manipulation. For example, a table is included in the output when the human is relocating a spam container on the table, while the table is ignored for the fridge opening task. These results suggest that the scene analyzer encodes the scene information with respect to the human’s actions. We prompted GPT-4V to explain the results of the object selection process and the reasons behind those choices. In practice, we found this approach resulted in reasonable outputs. Finally, based on the given text instructions and environmental information, the task planner outputs a sequence of tasks Wake et al. (2023c).
接下来，场景分析器会根据指令和视频数据的第一帧（或环境图像）将预期工作环境编译为文本信息。这些环境信息包括 GPT-4V 识别的物体名称列表、物体的可抓取属性以及物体间的空间关系。尽管这些计算过程是 GPT-4V 内部的黑箱操作，但信息输出基于 GPT-4V 的知识库和图像/文本输入。图 17 展示了我们场景分析器的示例输出结果。如图所示，GPT-4V 能成功筛选出与操作相关的物体。例如当人类移动桌面的垃圾箱时，输出结果包含桌子；而在执行冰箱开门任务时，桌子信息则被忽略。这些结果表明场景分析器会根据人类动作来编码场景信息。我们提示 GPT-4V 解释物体选择过程的结果及其决策依据，实践中发现该方法能产生合理输出。最后，基于给定的文本指令和环境信息，任务规划器会输出一系列任务序列（Wake 等人，2023c）。

Image-language understanding is a task that involves the interpretation of visual content in a given image with language and the generation of associated linguistic descriptions. This task is critical to the development of AI agents that can interact with the world in a more human-like manner. Some of most popular ones are image captioning Lin et al. (2014); Sharma et al. (2018); Young et al. (2014); Krishna et al. (2016), referring expression Yu et al. (2016); Karpathy et al. (2014), and visual question answering Antol et al. (2015); Ren et al. (2015); Singh et al. (2019).
图像-语言理解是指通过语言解释给定图像中的视觉内容并生成相关文字描述的任务。该任务对于开发能以更类人方式与世界交互的 AI 智能体至关重要。其中最典型的任务包括：图像描述生成（Lin 等人，2014；Sharma 等人，2018；Young 等人，2014；Krishna 等人，2016）、指代表达（Yu 等人，2016；Karpathy 等人，2014）以及视觉问答（Antol 等人，2015；Ren 等人，2015；Singh 等人，2019）。

More recently, knowledge-intensive Visual Question Answering tasks such as OKVQA Marino et al. (2019), KB-VQA Wang et al. (2015), FVQA Wang et al. (2017), and WebQA Chang et al. (2021) have been introduced. Multimodal agents should capable of identifying objects in an image, comprehending their spatial relationships, generating accurate descriptive sentences about the scene, and utilizing reasoning skills to handle knowledge-intensive visual reasoning. This requires not just object recognition capabilities, but also a deep understanding of spatial relationships, visual semantics, and the ability to map these visual elements to linguistic constructs with integration of the world knowledge.
近年来，研究者提出了知识密集型视觉问答任务，如 OKVQA（Marino 等人，2019）、KB-VQA（Wang 等人，2015）、FVQA（Wang 等人，2017）和 WebQA（Chang 等人，2021）。多模态智能体应当具备识别图像中的物体、理解其空间关系、生成准确场景描述句以及运用推理技能处理知识密集型视觉推理的能力。这不仅需要物体识别能力，还需要深刻理解空间关系、视觉语义，并能结合世界知识将这些视觉元素映射到语言结构中。

• •

  Knowledge-Intensive Models: As introduced in INK Park et al. (2022), and KAT Gui et al. (2022a), an intensive neural knowledge task that incorporates required knowledge annotated by humans to support knowledge-intensive retrieval task.
  知识密集型模型：如 INK Park 等人（2022 年）和 KAT Gui 等人（2022a 年）所提出的，这是一种融入人类标注的必要知识以支持知识密集型检索任务的密集型神经知识任务。

• •

  Multimodal-Agents: There has been a growing interest in multimodal language models like Chameleon Lu et al. (2023) and MM-React Yang et al. (2023c).
  多模态智能体：像 Chameleon Lu 等人（2023）和 MM-React Yang 等人（2023c）这样的多模态语言模型正受到越来越多的关注。

• •

  Visual Instruction Tuning: VCLGui et al. (2022b), Mini-GPT4 Zhu et al. (2023), MPLUG-OWL Ye et al. (2023b), LSKD Park et al. (2023c) generate image-level instruction tuning dataset.
  视觉指令调优：VCLGui 等人（2022b）、Mini-GPT4 Zhu 等人（2023）、MPLUG-OWL Ye 等人（2023b）、LSKD Park 等人（2023c）生成了图像级别的指令调优数据集。

Recognizing task directives and taking action has been a fundamental challenge in interactive AI and natural language processing for decades. With the recent advances in deep learning, there is a growing interest in studying these areas jointly to improve human-agent collaboration. We identify three specific directions, among others, to improve language-grounded agents:
数十年来，识别任务指令并采取相应行动始终是交互式人工智能和自然语言处理领域的核心挑战。随着深度学习的最新进展，联合研究这些领域以提升人机协作效能正引发日益广泛的关注。我们特别提出三个改进语言基础智能体的发展方向：

• •

  Tool use and querying from knowledge bases. This direction emphasizes the importance of integrating external knowledge bases, web search, or other helpful tools into the reasoning processes of AI agents. By leveraging structured and unstructured data from various sources, agents can enhance their understanding and provide more accurate and context-aware responses. Furthermore, it fosters the agent’s ability to proactively seek out information when faced with unfamiliar scenarios or queries, ensuring more comprehensive and informed responses. Examples include Toolformer Schick et al. (2023) and Retrieve What You Need Wang et al. (2023g).
  工具调用与知识库查询。该方向强调将外部知识库、网络搜索或其他辅助工具整合至 AI 智能体推理流程的重要性。通过利用来自多源的结构化与非结构化数据，智能体可深化理解能力，提供更精准且情境感知的响应。此外，这能增强智能体在遭遇陌生场景或查询时主动获取信息的能力，确保回应的全面性与可靠性。典型案例如 Schick 等人(2023)的 Toolformer 和王等人(2023g)的 Retrieve What You Need。

• •

  Improved agent reasoning and planning. Enhancing the agent’s ability to reason and plan is pivotal for effective human-agent collaboration. This involves the development of models that can understand complex instructions, infer user intentions, and predict potential future scenarios. This can be accomplished by asking the agent to reflect on past actions and failures as in ReAct Yao et al. (2023a), or by structuring the agent thought process as a form of search Yao et al. (2023b). By simulating different outcomes and assessing the ramifications of various actions, agents can make more informed context-aware decisions.
  改进智能体推理与规划能力。提升智能体的推理与规划能力对于实现高效人机协作至关重要，这需要开发能够理解复杂指令、推断用户意图并预测未来潜在场景的模型。实现路径包括让智能体像 ReAct(Yao 等人，2023a)研究那样反思过往行动与失败经验，或是如 Yao 等人(2023b)提出的将智能体思维过程构建为搜索机制。通过模拟不同结果并评估各类行动的潜在影响，智能体能够做出更具环境感知力的明智决策。

• •

  Incorporating system and human feedback. AI agents can frequently operate in two primary contexts: environments that provide explicit signals about the effectiveness of their actions (system feedback), and settings where they collaborate with humans who can offer verbal critiques (human feedback). This direction underscores the need for adaptive learning mechanisms that allow agents to refine their strategies and rectify mistakes, such as in AutoGen Wu et al. (2023). The ability to continuously learn and adapt from diverse feedback sources ensures that agents remain helpful and aligned for user needs.
  融入系统和人类反馈。AI 智能体通常可在两种主要情境下运作：一种是环境提供明确行动效果信号（系统反馈），另一种是与能提供口头评价的人类协作（人类反馈）。这一研究方向强调需要自适应学习机制，使智能体能够优化策略并纠正错误，例如 AutoGen Wu 等人（2023）所示例。通过持续从多元反馈源学习与适应的能力，确保智能体始终贴合用户需求并保持实用性。

Recognizing and understanding agent content and natural language has been a fundamental challenge in interactive AI and natural language processing for decades. With the recent advances in deep learning, there is a growing interest in studying these two areas jointly for deep understanding of both agent planning or human feedback for knowledge-inference and natural language generation. These are the key components of many human-machine-interaction agents, such as “AutoGen"Wu et al. (2023) and “Retrieve What You Need"Wang et al. (2023g).
数十年来，识别和理解智能体内容与自然语言一直是交互式人工智能和自然语言处理领域的核心挑战。随着深度学习技术的最新突破，学界日益关注将这两个领域相结合进行深入研究，以期实现智能体规划/人类知识推理反馈与自然语言生成的深度融合。这些技术构成了诸多"人机交互智能体"（如吴等人 2023 年提出的"AutoGen"与王等人 2023g 提出的"Retrieve What You Need"）的关键组件。

Furthermore, the creation of LLM Agents that can be trained to effectively follow human instructions has become an important area of research. Initial models used human feedback to train a proxy reward model to simulate human preferences, through a process known as Reinforcement Learning with Human Feedback (RLHF) Ouyang et al. (2022). This process produced models such as InstructGPT and ChatGPT. In order to more efficiently train instruction-following LLM agents without needing human labels, researchers developed a more efficient method for instruction-tuning that trains the LLM agent directly on instruction/response pairs, either generated by humans like Dolly 2.0 ⁵⁵5Dolly 2.0 blogpost link or automatically from LLMs like Alpaca Taori et al. (2023). We show the overall Alpaca training pipeline in Figure 28.
此外，训练能够有效遵循人类指令的 LLM 智能体已成为重要研究领域。早期模型通过人类反馈训练代理奖励模型来模拟人类偏好，这一过程被称为基于人类反馈的强化学习（RLHF）（Ouyang 等人，2022）。该方法催生了 InstructGPT 和 ChatGPT 等模型。为更高效地训练指令跟随型 LLM 智能体而无需人工标注，研究者开发了更高效的指令微调方法——直接使用指令/响应对训练 LLM 智能体，这些数据对或由人类生成（如 Dolly 2.0 ⁵⁵5Dolly 2.0 blogpost link ），或通过 LLMs 自动生成（如 Alpaca，Taori 等人，2023）。图 28 展示了 Alpaca 的整体训练流程。

Despite the growing adoption of conversational and self-feedback systems, these forms of AI still do not perform well with regard to generating factually correct responses from their own implicit knowledge and therefore often use external tools like web search and knowledge retrieval mechanisms at inference-time to augment their response as a consequence. Addressing this would help create more engaging experiences for users in many real-life applications. In social conversations (such as those on social media platforms like Instagram and Facebook), or with Q+A websites (such as Ask or Quora), people usually engage with others through a series of comments and by web-searching for information and knowledge relevant to the discussion. Thus, the task of generating conversational turns in this context is not to simply bootstrap upon traditional NLP models and tasks, but to use agents to generate dialogue through intelligent behaviors that reflect knowledge search and acquisition Peng et al. (2023). In this way, intelligent agents for NLP tasks extends the task description and improves upon the interpretability of the response by adding an explicit knowledge search and retrieval step during dialogue. Incorporating these web search and retrieval agents as feedback during dialogue will help to engage further and deeper the social interactions between humans and agents Wang et al. (2023e). As the Fig 29 showed, we introduced a new modeling paradigm for transformer language models that detects and extracts important logical structures and information from input texts and then integrates them into the input embeddings through carefully designed multi-layer hierarchical logical projections to infuse logical structures into pre-trained language models as one kind of NLP agent. Wang et al. (2023e) propose a novel approach to construct logic-aware input embeddings for transformer language models through a combination of logic detection, logic mapping and hierarchical logical projections, and then develop a corresponding new modeling paradigm that can upgrade all existing transformer language models into logical transformers to consistently boost their performance. The proposed logical transformer agent consistently achieve superior performance over their baseline transformer models through a deeper understanding of the logical structures of texts. To human users, it is often these aspects that are more important for delivering a meaningful and interesting conversation via a agent-based coordination between dialogue and information retrieval. Delving deep into natural language processing, this topic will discuss the advancements and leader-board in making LLMs more agentic and better suited for various language-centered tasks.
尽管对话式系统和自我反馈系统的应用日益广泛，但这类人工智能在基于自身隐性知识生成事实性正确回应方面仍表现欠佳，因此常在推理阶段依赖网络搜索和知识检索机制等外部工具来增强响应能力。解决这一问题将有助于在诸多现实应用中为用户创造更具吸引力的交互体验。在社交媒体平台（如 Instagram 和 Facebook）的社交对话，或问答网站（如 Ask 和 Quora）的互动中，人们通常通过系列评论和网络信息检索参与讨论。因此，这类场景下的对话轮次生成任务并非简单套用传统 NLP 模型，而是需要智能体通过反映知识搜索与获取的智能行为来产生对话（Peng 等，2023）。这种面向 NLP 任务的智能体通过显式增加对话过程中的知识检索步骤，既扩展了任务描述范畴，又提升了响应结果的可解释性。将网络搜索与检索智能体作为对话反馈机制，能促进人类与智能体之间更深入的社会化互动（Wang 等，2023e）。如图 29 所示，我们提出了一种新型 Transformer 语言模型建模范式：通过检测并提取输入文本中的关键逻辑结构与信息，再经精心设计的层级逻辑投影将其融入输入嵌入，从而将逻辑结构注入预训练语言模型作为 NLP 智能体的一种实现。Wang 等（2023e）提出了通过逻辑检测、逻辑映射和层级逻辑投影构建逻辑感知输入嵌入的新方法，继而开发出可将现有 Transformer 语言模型升级为逻辑 Transformer 的新型建模范式，持续提升模型性能。这种逻辑 Transformer 智能体凭借对文本逻辑结构的深层理解，始终优于基线 Transformer 模型。对人类用户而言，正是这种基于智能体的对话-信息检索协同机制，对实现有意义且有趣的对话至关重要。本专题将深入探讨自然语言处理领域如何通过增强 LLMs 的智能体特性，使其更胜任各类以语言为中心的任务，并分析相关技术进展与性能排行榜。

An open-domain question answering (QA) system usually follows a retrieve-then-read paradigm, in which a retriever is used to retrieve relevant passages from a large corpus, and then a reader generates answers based on the retrieved passages and the original question. In  Wang et al. (2023g), we propose a simple and novel mutual learning framework to improve the performance of retrieve-then-read-style models via an intermediate module named the knowledge selector agent, which we train with reinforcement learning. The fine-grained knowledge selector into the retrieve-then-reader paradigm, whose goal is to construct a small subset of passages which retain question-relevant information. As showed in Figure 30, The knowledge selector agent is trained as a component of our novel mutual learning framework, which iteratively trains the knowledge selector and the reader. We adopt a simple and novel approach employing policy gradients to optimize the knowledge selector agnet, using feedback from the reader to train it to select a small and informative set of passages. This approach avoids brute-force search or manually-designed heuristics, without requiring any annotated query-document pairs for supervision. We show that iteratively training the reader and the knowledge selector agent leads to better predictive performance on some public open-domain question answering benchmarks.
开放域问答系统通常采用"检索-阅读"范式：先通过检索器从大规模语料库获取相关段落，再由阅读器基于检索结果和原始问题生成答案。在 Wang 等（2023g）的研究中，我们提出了一种新颖的互学习框架，通过名为"知识选择器智能体"的中间模块（采用强化学习训练）来提升该范式的性能。该细粒度知识选择器旨在构建保留问题相关信息的精简段落子集。如图 30 所示，知识选择器智能体作为互学习框架的组件，与阅读器进行迭代训练。我们采用策略梯度法优化知识选择器，利用阅读器的反馈训练其选择信息量高的小规模段落集合。这种方法避免了暴力搜索或人工设计启发式规则，且无需监督式的查询-文档标注对。实验表明，阅读器与知识选择器智能体的迭代训练能有效提升多个公开开放域问答基准的预测性能。

For our competition, we will release a benchmark, the CuisineWorld benchmark, which includes a text interface that includes extendable task definition files, and an interface for multi-agent interaction, and human-machine interactions. We introduce the gaming interaction task in which the goal is to generate relevant, appropriate, multi-agent collaboration strategies that can maximize collaboration efficiency. We evaluate the collaboration efficiency with the proposed evaluation metric: CoS.
竞赛将发布 CuisineWorld 基准测试，包含可扩展任务定义的文本接口、多智能体交互接口及人机交互接口。我们设计了游戏交互任务，目标是生成能最大化协作效率的相关多智能体策略，并使用创新评估指标 CoS 衡量协作效率。

The “CuisineWorld" dataset was collected by Microsoft, UCLA, and Stanford University. The goal of the competition is to explore how different, existing and novel, grounded-LLM and interactive techniques perform with this benchmark and establish strong baselines for the task of multi-agent gaming infrastructure.
该数据集由微软、加州大学洛杉矶分校和斯坦福大学联合采集，旨在探索现有及新型具身 LLM 技术在此基准测试中的表现，为多智能体游戏基础设施建立强基线。

The dataset of CuisineWorld includes:
数据集包含：

- A selection of well-defined multi-agent collaboration tasks.
- 精选的多智能体协作任务
- An API system to facilitate agent interactions.
- 支持智能体交互的 API 系统
- An automatic evaluation system.
- 自动评估系统

(The link for downloading the dataset will soon be made available and this article will be updated to include it here.)
（数据集下载链接即将公布，本文档将同步更新）

• •

  We provide a dataset and related the benchmark, called Microsoft MindAgent and and correspondingly release a dataset “CuisineWorld” to the to the research community.
  我们发布名为 Microsoft MindAgent 的基准测试及配套"CuisineWorld"数据集，向研究社区开放。

• •

  We will provide benchmarks to evaluate and rank the submitted “MindAgent" algorithms. We will also provide baseline results generated using popular infrastructures.
  将提供基准来评估提交的"MindAgent"算法并排名，同时公布基于主流基础设施生成的基线结果。

The quality of multi-agent collaboration efficiency is determined by the new “cos" auto-metric (from MindAgent Gong et al. (2023a)). The final rating of out metric is calculated as an average over the evaluated collaboration efficiency metrics of the multi-agent system on all tasks. Human evaluators will be asked to rate individual responses as well as provide subjective judgement of the engagement, breadth and an overall quality of the users’ interactions with the agents.
多智能体协作效率的质量由新型"cos"自动度量标准决定（源自 MindAgent Gong 等人(2023a)的研究）。最终评分将计算为多智能体系统在所有任务上协作效率指标的平均值。人类评估者将被要求对单个响应进行评分，并对用户与智能体交互的参与度、广度及整体质量提供主观判断。

• •

  Automated Evaluation. We plan to release a leaderboard, starting on the release date (TBA), registered participants will be asked to submit their results on the task associated with the dataset “CuisineWorld" (our publicly released dataset for the leaderboard). Submission of results will be closed on the end date (TBA). Each team will be required to submit their generated results on the testing set for automated evaluation of the “cos" metric.
  自动评估。我们计划在发布日期（待定）启动排行榜，要求注册参与者提交其在"CuisineWorld"数据集（我们为排行榜公开发布的专用数据集）相关任务上的结果。结果提交将在截止日期（待定）关闭。每个团队需提交测试集上的生成结果，用于"cos"指标的自动化评估。

• •

  Human Evaluation on our leaderboard. The leaderboard participants will need to provide a submission file generated by evaluation scripts locally. We will use the evalAI system to check the submission file and optionally rerun the code for top challenge contenders. Therefore, teams must also submit their code with a Readme file on how to run their code. Human evaluation will be performed by the organization team.
  排行榜人工评估。参与者需提供本地通过评估脚本生成的提交文件。我们将使用 evalAI 系统核验提交文件，并对顶尖参赛者选择性重运行代码。因此各团队必须同时提交代码及说明运行方式的 Readme 文件。人工评估将由组委会执行。

• •

  Winner Announcement. We will make an announcement of the winners and post the final ratings of the submissions on our leaderboard.
  优胜者公告。我们将公布获胜团队名单，并在排行榜上发布所有提交作品的最终评分。