Figure 1. We present SAM-6D for zero-shot 6D object pose estimation. SAM-6D takes an RGB image (a) and a depth map (b) of a cluttered scene as inputs, and performs instance segmentation (d) and pose estimation (e) for novel objects ©. We present the qualitative results of SAM-6D on the seven core datasets of the BOP benchmark [54], including YCB-V, LM-O, HB, T-LESS, IC-BIN, ITODD and TUD-L, arranged from left to right. Best view in the electronic version.
图 1. 我们提出了用于零样本 6D 物体姿态估计的 SAM-6D。SAM-6D 以杂乱场景的 RGB 图像(a)和深度图(b)作为输入，执行新颖物体©的实例分割(d)和姿态估计(e)。我们展示了 SAM-6D 在 BOP 基准测试[54]的七个核心数据集上的定性结果，包括 YCB-V、LM-O、HB、T-LESS、IC-BIN、ITODD 和 TUD-L，图中从左到右排列。电子版中效果最佳。

stance segmentation and pose estimation. Given the target objects, SAM-6D employs two dedicated sub-networks, namely Instance Segmentation Model (ISM) and Pose Estimation Model (PEM), to perform these steps on cluttered RGB-D images. ISM takes SAM as an advanced starting point to generate all possible object proposals and selectively preserves valid ones through meticulously crafted object matching scores in terms of semantics, appearance and geometry. By treating pose estimation as a partial-topartial point matching problem, PEM performs a two-stage point matching process featuring a novel design of background tokens to construct dense 3D-3D correspondence,
姿态分割与姿态估计。针对目标物体，SAM-6D 采用两个专用子网络，即实例分割模型（ISM）和姿态估计模型（PEM），在杂乱的 RGB-D 图像上执行这些步骤。ISM 以 SAM 作为先进的起点，生成所有可能的物体提议，并通过精心设计的语义、外观和几何对象匹配分数，有选择地保留有效提议。PEM 将姿态估计视为部分到部分的点匹配问题，执行两阶段点匹配过程，采用新颖的背景标记设计来构建密集的 3D-3D 对应关系，
ultimately yielding the pose estimates. Without bells and whistles, SAM-6D outperforms the existing methods on the seven core datasets of the BOP Benchmark for both instance segmentation and pose estimation of novel objects.
最终得出姿态估计结果。无需复杂技巧，SAM-6D 在 BOP 基准的七个核心数据集上，在新颖物体的实例分割和姿态估计方面均优于现有方法。

Object pose estimation is fundamental in many real-world applications, such as robotic manipulation and augmented reality. Its evolution has been significantly influenced by the emergence of deep learning models. The most studied task in this field is Instance-level 6D Pose Estimation [18, 19, 51, 58, 60, 63], which demands annotated training images of the target objects, thereby making the deep models object-specific. Recently, the research emphasis gradually shifts towards the task of Category-level 6D Pose Estimation [7, 29-32, 56, 61] for handling unseen objects, yet provided they belong to certain categories of interest. In this paper, we thus delve into a broader task setting of Zero-shot 6D Object Pose Estimation [5, 28], which aspires to detect all instances of novel objects, unseen during training, and estimate their 6D poses. Despite its significance, this zeroshot setting presents considerable challenges in both object detection and pose estimation.
物体姿态估计在许多现实应用中具有基础性作用，如机器人操作和增强现实。深度学习模型的出现极大地推动了该领域的发展。该领域研究最多的任务是实例级 6D 姿态估计[18, 19, 51, 58, 60, 63]，该任务需要目标物体的带注释训练图像，因此使得深度模型具有针对特定物体的特性。近年来，研究重点逐渐转向类别级 6D 姿态估计[7, 29-32, 56, 61]，以处理未见过的物体，但这些物体属于某些感兴趣的类别。在本文中，我们进一步探讨了更广泛的零样本 6D 物体姿态估计任务[5, 28]，该任务旨在检测训练期间未见过的全新物体实例，并估计它们的 6D 姿态。尽管该零样本设置意义重大，但在物体检测和姿态估计方面都面临着相当大的挑战。

Recently, Segment Anything Model (SAM) [26] has garnered attention due to its remarkable zero-shot segmentation performance, which enables prompt segmentation with a variety of prompts, e.g., points, boxes, texts or masks. By prompting SAM with evenly sampled 2D grid points, one can generate potential class-agnostic object proposals, which may be highly beneficial for zero-shot 6D object pose estimation. To this end, we propose a novel framework, named SAM-6D, which employs SAM as an advanced starting point for the focused zero-shot task. Fig. 2 gives an overview illustration of SAM-6D. Specifically, SAM-6D employs an Instance Segmentation Model (ISM) to realize instance segmentation of novel objects by enhancing SAM with a carefully crafted object matching score, and a Pose Estimation Model (PEM) to solve object poses through a two-stage process of partial-to-partial point matching.
最近，Segment Anything Model（SAM）[26] 因其出色的零样本分割性能而受到关注，该性能使得通过多种提示（如点、框、文本或掩码）实现快速分割成为可能。通过使用均匀采样的二维网格点对 SAM 进行提示，可以生成潜在的类别无关的物体提议，这对于零样本 6D 物体姿态估计可能非常有益。为此，我们提出了一个新颖的框架，命名为 SAM-6D，该框架将 SAM 作为专注于零样本任务的先进起点。图 2 展示了 SAM-6D 的整体示意图。具体而言，SAM-6D 采用实例分割模型（ISM）通过引入精心设计的物体匹配分数来增强 SAM，实现对新颖物体的实例分割；同时采用姿态估计模型（PEM）通过部分到部分点匹配的两阶段过程来解决物体姿态问题。

The Instance Segmentation Model (ISM) is developed using SAM to take advantage of its zero-shot abilities for generating all possible class-agnostic proposals, and then assigns a meticulously calculated object matching score to each proposal for ascertaining whether it aligns with a given novel object. In contrast to methods that solely focus on object semantics [5, 40], we design the object matching scores considering three terms, including semantics, appearance and geometry. For each proposal, the first term assesses its semantic matching degree to the rendered templates of the object, while the second one further evaluates its appearance similarities to the best-matched template. The final term considers the matching degree based on ge-
实例分割模型（ISM）是利用 SAM 开发的，旨在利用其零样本能力生成所有可能的类别无关提议，然后为每个提议分配一个精心计算的对象匹配分数，以确定其是否与给定的新颖对象相符。与仅关注对象语义的方法[5, 40]不同，我们设计的对象匹配分数考虑了三个方面，包括语义、外观和几何。对于每个提议，第一个方面评估其与对象渲染模板的语义匹配程度，第二个方面进一步评估其与最佳匹配模板的外观相似度。最后一个方面则考虑基于几何的匹配程度。

Figure 2. An overview of our proposed SAM-6D, which consists of an Instance Segmentation Model (ISM) and a Pose Estimation Model (PEM) for joint instance segmentation and pose estimation of novel objects in RGB-D images. ISM leverages the Segment Anything Model (SAM) [26] to generate all possible proposals and selectively retains valid ones based on object matching scores. PEM involves two stages of point matching, from coarse to fine, to establish 3D-3D correspondence and calculate object poses for all valid proposals. Best view in the electronic version.
图 2. 我们提出的 SAM-6D 概述，该方法由实例分割模型（ISM）和姿态估计模型（PEM）组成，用于 RGB-D 图像中新颖物体的联合实例分割和姿态估计。ISM 利用 Segment Anything Model（SAM）[26]生成所有可能的候选区域，并根据物体匹配分数有选择地保留有效候选。PEM 包括两个阶段的点匹配，从粗到细，建立 3D-3D 对应关系并计算所有有效候选的物体姿态。电子版中效果最佳。
ometry, such as object shape and size, by calculating the Intersection-over-Union (IoU) value between the bounding boxes of the proposal and the 2D projection of the object transformed by a rough pose estimate.
通过计算候选框与由粗略姿态估计变换后的物体二维投影之间的交并比（IoU）值，来估计几何信息，如物体形状和大小。

The Pose Estimation Model (PEM) is designed to calculate a 6D object pose for each identified proposal that matches the novel object. Initially, we formulate this pose estimation challenge as a partial-to-partial point matching problem between the sampled point sets of the proposal and the target object, considering the factors such as occlusions, segmentation inaccuracies, and sensor noises. To solve this problem, we propose a simple yet effective solution that involves the use of background tokens; specifically, for the two point sets, we learn to align their non-overlapped points with the background tokens in the feature space, and thus effectively establish an assignment matrix to build the necessary correspondence for predicting the object pose. Based on the design of background tokens, we further develop PEM with two point matching stages, i.e., Coarse Point Matching and Fine Point Matching. The first stage realizes sparse correspondence to derive an initial object pose, which is subsequently used to transform the point set of the proposal, enabling the learning of positional encodings. The second stage incorporates the positional encodings of the two point sets to inject the initial correspondence, and
姿态估计模型（PEM）旨在为每个与新颖物体匹配的识别提议计算 6D 物体姿态。最初，我们将此姿态估计问题表述为提议点集与目标物体采样点集之间的部分到部分点匹配问题，考虑了遮挡、分割不准确和传感器噪声等因素。为了解决该问题，我们提出了一种简单而有效的解决方案，涉及使用背景标记；具体来说，对于这两个点集，我们学习在特征空间中将它们未重叠的点与背景标记对齐，从而有效建立分配矩阵，构建预测物体姿态所需的对应关系。基于背景标记的设计，我们进一步开发了包含两个点匹配阶段的 PEM，即粗匹配和细匹配。第一阶段实现稀疏对应以推导初始物体姿态，随后利用该姿态变换提议点集，从而实现位置编码的学习。 第二阶段结合了两组点集的位置编码，以注入初始对应关系，并
builds dense correspondence for estimating a more precise object pose. To effectively model dense interactions in the second stage, we propose an innovative design of Sparse-toDense Point Transformers, which realize interactions on the sparse versions of the dense features, and subsequently, distribute the enhanced sparse features back to the dense ones using Linear Transformers [12, 24].
构建密集对应关系以估计更精确的物体姿态。为了在第二阶段有效建模密集交互，我们提出了一种创新设计的稀疏到密集点变换器（Sparse-to-Dense Point Transformers），该设计在密集特征的稀疏版本上实现交互，随后利用线性变换器（Linear Transformers）[12, 24]将增强的稀疏特征分布回密集特征。

For the two models of SAM-6D, ISM, built on SAM, does not require any network re-training or fine-tuning, while PEM is trained on the large-scale synthetic images of ShapeNet-Objects [4] and Google-Scanned-Objects [9] datasets provided by [28]. We evaluate SAM-6D on the seven core datasets of the BOP benchmark [54], including LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, and YCBV. The qualitative results are visualized in Fig. 1. SAM-6D outperforms the existing methods on both tasks of instance segmentation and pose estimation of novel objects, thereby showcasing its robust generalization capabilities.
对于 SAM-6D 的两个模型，基于 SAM 构建的 ISM 不需要任何网络重新训练或微调，而 PEM 则在[28]提供的 ShapeNet-Objects [4]和 Google-Scanned-Objects [9]数据集的大规模合成图像上进行训练。我们在 BOP 基准测试[54]的七个核心数据集上评估了 SAM-6D，包括 LM-O、T-LESS、TUD-L、IC-BIN、ITODD、HB 和 YCBV。定性结果如图 1 所示。SAM-6D 在新颖物体的实例分割和姿态估计两个任务上均优于现有方法，展示了其强大的泛化能力。

Our main contributions could be summarized as follows:
我们的主要贡献可总结如下：

Segment Anything (SA) [26], is a promptable segmentation task that focuses on predicting valid masks for various types of prompts, e.g., points, boxes, text, and masks. To tackle this task, the authors propose a powerful segmentation model called Segment Anything Model (SAM), which comprises three components, including an image encoder, a prompt encoder and a mask decoder. SAM has demonstrated remarkable zero-shot transfer segmentation performance in real-world scenarios, including challenging situations such as medical images [36, 37, 71], camouflaged objects [22, 55], and transparent objects [13, 23]. Moreover, SAM has exhibited high versatility across numerous vision applications [69], such as image inpainting [35, 62, 64, 67], object tracking [15, 65, 73], 3D detection and segmentation [2, 66, 70], and 3D reconstruction [3, 49, 59].
任意分割（Segment Anything，SA）[26] 是一项可提示的分割任务，重点在于预测针对各种类型提示（如点、框、文本和掩码）的有效掩码。为了解决该任务，作者提出了一种强大的分割模型，称为任意分割模型（Segment Anything Model，SAM），该模型包含三个部分：图像编码器、提示编码器和掩码解码器。SAM 在真实场景中展现了卓越的零样本迁移分割性能，包括医学图像[36, 37, 71]、伪装物体[22, 55]和透明物体[13, 23]等挑战性情况。此外，SAM 在众多视觉应用中表现出高度的通用性[69]，例如图像修复[35, 62, 64, 67]、目标跟踪[15, 65, 73]、三维检测与分割[2, 66, 70]以及三维重建[3, 49, 59]。

Recent studies have also investigated semantically segmenting anything due to the critical role of semantics in vision tasks. Semantic Segment Anything (SSA) [6] is proposed on top of SAM, aiming to assign semantic categories to the masks generated by SAM. Both PerSAM [72] and Matcher [34] employ SAM to segment the object belonging to a specific category in a query image by searching for point prompts with the aid of a reference image containing an object of the same category. CNOS [40] is proposed to segment all instances of a given object model, which firstly generates mask proposals via SAM and subsequently filters out proposals with low feature similarities against object templates rendered from the object model.
最近的研究也探讨了语义分割任意物体，原因在于语义在视觉任务中的关键作用。语义分割任意物体（Semantic Segment Anything，SSA）[6]是在 SAM 基础上提出的，旨在为 SAM 生成的掩码分配语义类别。PerSAM [72]和 Matcher [34]都利用 SAM，通过参考图像中包含同类别物体的点提示搜索，在查询图像中分割属于特定类别的物体。CNOS [40]被提出用于分割给定物体模型的所有实例，其首先通过 SAM 生成掩码提议，随后过滤掉与从物体模型渲染的物体模板特征相似度较低的提议。

For efficiency, FastSAM [74] is proposed by utilizing instance segmentation networks with regular convolutional networks instead of visual transformers used in SAM. Additionally, MobileSAM [68] replaces the heavy encoder of SAM with a lightweight one through decoupled distillation.
为了提高效率，FastSAM [74]通过使用常规卷积网络的实例分割网络替代 SAM 中使用的视觉变换器而被提出。此外，MobileSAM [68]通过解耦蒸馏将 SAM 的重型编码器替换为轻量级编码器。

Methods Based on Image Matching Methods within this group [1, 28, 33, 38, 39, 41, 42, 46, 50] often involve comparing object proposals to templates of the given novel objects, which are rendered with a series of object poses, to retrieve the best-matched object poses. For example, Gen6D [33], OVE6D [1], and GigaPose [41] are designed to select the viewpoint rotations via image matching and then estimate the in-plane rotations to obtain the final estimates. MegaPose [28] employs a coarse estimator to treat image matching as a classification problem, of which the recognized object poses are further updated by a refiner.
基于图像匹配的方法 该组方法[1, 28, 33, 38, 39, 41, 42, 46, 50]通常涉及将目标提议与给定新颖物体的模板进行比较，这些模板通过一系列物体姿态渲染，以检索最佳匹配的物体姿态。例如，Gen6D [33]、OVE6D [1]和 GigaPose [41]设计用于通过图像匹配选择视角旋转，然后估计平面内旋转以获得最终估计。MegaPose [28]采用粗略估计器，将图像匹配视为分类问题，识别出的物体姿态随后由细化器进一步更新。
Methods Based on Feature Matching Methods within this group [5, 10, 11, 17, 20, 53] align the 2D pixels or 3D points of the proposals with the object surface in the feature space [21, 52], thereby building correspondence to compute object poses. OnePose [53] matches the pixel descriptors of proposals with the aggregated point descriptors of the point sets constructed by Structure from Motion (SfM) for 2D-3D correspondence, while OnePose++ [17] further improves it with a keypoint-free SfM and a sparse-to-dense 2D-3D matching model. ZeroPose [5] realizes 3D-3D matching via geometric structures, and GigaPose [41] establishes 2D-2D correspondence to regress in-plane rotation and 2D scale. Moreover, [11] introduces a zero-shot category-level 6D pose estimation task, along with a self-supervised semantic correspondence learning method. Unlike the above one-stage point matching work, the unique contributions in our Pose Estimation Model are: (a) a two-stage pipeline that boosts performance by incorporating coarse correspondence for finer matching, (b) an efficient design of background tokens to eliminate the need of optimal transport with iterative optimization [48], and © a Sparse-to-Dense Point Transformer to effectively model dense relationship.
基于特征匹配的方法 该组方法[5, 10, 11, 17, 20, 53]通过在特征空间[21, 52]中将提议的 2D 像素或 3D 点与物体表面对齐，从而建立对应关系以计算物体姿态。OnePose[53]通过将提议的像素描述符与由结构光束法（SfM）构建的点集的聚合点描述符进行匹配，实现 2D-3D 对应；而 OnePose++[17]则通过无关键点的 SfM 和稀疏到稠密的 2D-3D 匹配模型进一步提升了性能。ZeroPose[5]通过几何结构实现 3D-3D 匹配，GigaPose[41]建立 2D-2D 对应以回归平面内旋转和 2D 尺度。此外，[11]引入了零样本类别级 6D 姿态估计任务，并提出了一种自监督的语义对应学习方法。 与上述单阶段点匹配工作不同，我们的姿态估计模型的独特贡献在于：（a）一个两阶段流程，通过引入粗略对应来提升更精细匹配的性能，（b）一种高效的背景令牌设计，消除了使用迭代优化的最优传输[48]的需求，以及（c）一个稀疏到密集点变换器，有效建模密集关系。

We present SAM-6D for zero-shot 6D object pose estimation, which aims to detect all instances of a specific novel object, unseen during training, along with their 6D object poses in the RGB-D images. To realize the challenging task, SAM-6D breaks it down into two steps via two dedicated sub-networks, i.e., an Instance Segmentation Model (ISM) and a Pose Estimation Model (PEM), to first segment all instances and then individually predict their 6D poses, as shown in Fig. 2. We detail the architectures of ISM and PEM in Sec. 3.1 and Sec. 3.2, respectively.
我们提出了 SAM-6D 用于零样本 6D 物体姿态估计，旨在检测训练时未见过的特定新颖物体的所有实例及其在 RGB-D 图像中的 6D 物体姿态。为实现这一具有挑战性的任务，SAM-6D 通过两个专用子网络将其分解为两个步骤，即实例分割模型（ISM）和姿态估计模型（PEM），先分割所有实例，再单独预测它们的 6D 姿态，如图 2 所示。我们分别在第 3.1 节和第 3.2 节详细介绍 ISM 和 PEM 的架构。

SAM-6D uses an Instance Segmentation Model (ISM) to segment the instances of a novel object $\mathcal{O}$$\mathcal{O}$O\mathcal{O}. Given a cluttered scene, represented by an RGB image $\mathcal{I}$$\mathcal{I}$I\mathcal{I}, ISM leverages the zero-shot transfer capabilities of Segment Anything Model (SAM) [26] to generate all possible proposals $\mathcal{M}$$\mathcal{M}$M\mathcal{M}. For each proposal $m\in \mathcal{M}$$m\in \mathcal{M}$m inMm \in \mathcal{M}, ISM calculates an object matching score $s_m$$s_m$s_(m)s_{m} to assess the matching degree between $m$$m$mm and $\mathcal{O}$$\mathcal{O}$O\mathcal{O} in terms of semantics, appearance, and geometry. The matched instances with $\mathcal{O}$$\mathcal{O}$O\mathcal{O} can then be identified by simply setting a matching threshold ${\delta }_m$${\delta }_m$delta_(m)\delta_{m}.
SAM-6D 使用实例分割模型（ISM）对新颖物体 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 进行实例分割。给定一个由 RGB 图像 $\mathcal{I}$$\mathcal{I}$I\mathcal{I} 表示的杂乱场景，ISM 利用 Segment Anything Model（SAM）[26] 的零样本迁移能力生成所有可能的提议 $\mathcal{M}$$\mathcal{M}$M\mathcal{M} 。对于每个提议 $m\in \mathcal{M}$$m\in \mathcal{M}$m inMm \in \mathcal{M} ，ISM 计算一个物体匹配分数 $s_m$$s_m$s_(m)s_{m} ，以评估 $m$$m$mm 和 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 在语义、外观和几何方面的匹配程度。然后，通过简单设置匹配阈值 ${\delta }_m$${\delta }_m$delta_(m)\delta_{m} ，即可识别与 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 匹配的实例。

In this subsection, we initially provide a brief review of SAM in Sec. 3.1.1 and then explain the computation of the object matching score $s_m$$s_m$s_(m)s_{m} in Sec. 3.1.2.
在本小节中，我们首先在第 3.1.1 节简要回顾 SAM，然后在第 3.1.2 节解释物体匹配分数 $s_m$$s_m$s_(m)s_{m} 的计算方法。

Given an RGB image $\mathcal{I}$$\mathcal{I}$I\mathcal{I}, Segment Anything Model (SAM) [26] realizes promptable segmentation with various types of prompts ${\mathcal{P}}_r,e.g$${\mathcal{P}}_r,e.g$P_(r),e.g\mathcal{P}_{r}, e . g., points, boxes, texts, or masks. Specifically, SAM consists of three modules, including an image encoder ${\mathrm{\Phi }}_{\text{Image}}$${\mathrm{\Phi }}_{\text{Image }}$Phi_("Image ")\Phi_{\text {Image }}, a prompt encoder ${\mathrm{\Phi }}_{\text{Prompt}}$${\mathrm{\Phi }}_{\text{Prompt }}$Phi_("Prompt ")\Phi_{\text {Prompt }}, and a mask decoder ${\mathrm{\Psi }}_{\text{Mask}}$${\mathrm{\Psi }}_{\text{Mask }}$Psi_("Mask ")\Psi_{\text {Mask }}, which could be formulated as follows:
给定一张 RGB 图像 $\mathcal{I}$$\mathcal{I}$I\mathcal{I} ，Segment Anything Model（SAM）[26] 实现了基于多种提示类型 ${\mathcal{P}}_r,e.g$${\mathcal{P}}_r,e.g$P_(r),e.g\mathcal{P}_{r}, e . g （点、框、文本或掩码）的可提示分割。具体来说，SAM 包含三个模块：图像编码器 ${\mathrm{\Phi }}_{\text{Image}}$${\mathrm{\Phi }}_{\text{Image }}$Phi_("Image ")\Phi_{\text {Image }} 、提示编码器 ${\mathrm{\Phi }}_{\text{Prompt}}$${\mathrm{\Phi }}_{\text{Prompt }}$Phi_("Prompt ")\Phi_{\text {Prompt }} 和掩码解码器 ${\mathrm{\Psi }}_{\text{Mask}}$${\mathrm{\Psi }}_{\text{Mask }}$Psi_("Mask ")\Psi_{\text {Mask }} ，其可表示为：

where $\mathcal{M}$$\mathcal{M}$M\mathcal{M} and $\mathcal{C}$$\mathcal{C}$C\mathcal{C} denote the predicted proposals and the corresponding confidence scores, respectively.
其中 $\mathcal{M}$$\mathcal{M}$M\mathcal{M} 和 $\mathcal{C}$$\mathcal{C}$C\mathcal{C} 分别表示预测的提议和相应的置信度分数。

To realize zero-shot transfer, one can prompt SAM with evenly sampled 2D grids to yield all possible proposals, which can then be filtered based on confidence scores, retaining only those with higher scores, and applied to NonMaximum Suppression to eliminate redundant detections.
为了实现零样本迁移，可以用均匀采样的二维网格对 SAM 进行提示，以生成所有可能的提议，然后根据置信度分数进行筛选，仅保留分数较高的提议，并应用非极大值抑制以消除冗余检测。

Given the proposals $\mathcal{M}$$\mathcal{M}$M\mathcal{M}, the next step is to identify the ones that are matched with a specified object $\mathcal{O}$$\mathcal{O}$O\mathcal{O} by assigning each proposal $m\in \mathcal{M}$$m\in \mathcal{M}$m inMm \in \mathcal{M} with an object matching score $s_m$$s_m$s_(m)s_{m}, which comprises three terms, each evaluating the matches in terms of semantics, appearance, and geometry, respectively.
给定提议 $\mathcal{M}$$\mathcal{M}$M\mathcal{M} ，下一步是通过为每个提议 $m\in \mathcal{M}$$m\in \mathcal{M}$m inMm \in \mathcal{M} 分配一个物体匹配分数 $s_m$$s_m$s_(m)s_{m} 来识别与指定物体 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 匹配的提议，该分数包含三个部分，分别从语义、外观和几何三个方面评估匹配情况。

Following [40], we sample $N_{\mathcal{T}}$$N_{\mathcal{T}}$N_(T)N_{\mathcal{T}} object poses in $\mathrm{SE}(3)$$\mathrm{SE}(3)$SE(3)\operatorname{SE}(3) space to render the templates ${\left\{{\mathcal{T}}_k\right\}}_{k=1}^{N_{\mathcal{T}}}$${\left\{{\mathcal{T}}_k\right\}}_{k=1}^{N_{\mathcal{T}}}${T_(k)}_(k=1)^(N_(T))\left\{\mathcal{T}_{k}\right\}_{k=1}^{N_{\mathcal{T}}} of $\mathcal{O}$$\mathcal{O}$O\mathcal{O}, which are fed
根据[40]，我们在 $\mathrm{SE}(3)$$\mathrm{SE}(3)$SE(3)\operatorname{SE}(3) 空间中采样 $N_{\mathcal{T}}$$N_{\mathcal{T}}$N_(T)N_{\mathcal{T}} 个物体姿态，以渲染 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 的模板 ${\left\{{\mathcal{T}}_k\right\}}_{k=1}^{N_{\mathcal{T}}}$${\left\{{\mathcal{T}}_k\right\}}_{k=1}^{N_{\mathcal{T}}}${T_(k)}_(k=1)^(N_(T))\left\{\mathcal{T}_{k}\right\}_{k=1}^{N_{\mathcal{T}}} ，这些模板被输入
into a pre-trained visual transformer (ViT) backbone [8] of DINOv2 [45], resulting in the class embedding ${\mathit{f}}_{{\mathcal{T}}_k}^{cls}$${\mathit{f}}_{{\mathcal{T}}_k}^{cls}$f_(T_(k))^(cls)\boldsymbol{f}_{\mathcal{T}_{k}}^{c l s} and $N_{{\mathcal{T}}_k}^{\text{patch}}$$N_{{\mathcal{T}}_k}^{\text{patch }}$N_(T_(k))^("patch ")N_{\mathcal{T}_{k}}^{\text {patch }} patch embeddings ${\left\{{\mathit{f}}_{{\mathcal{T}}_k,i}^{\text{patch}}\right\}}_{i=1}^{N_{{\mathcal{T}}_k}^{\text{patch}}}$${\left\{{\mathit{f}}_{{\mathcal{T}}_k,i}^{\text{patch }}\right\}}_{i=1}^{N_{{\mathcal{T}}_k}^{\text{patch }}}${f_(T_(k),i)^("patch ")}_(i=1)^(N_(T_(k))^("patch "))\left\{\boldsymbol{f}_{\mathcal{T}_{k}, i}^{\text {patch }}\right\}_{i=1}^{N_{\mathcal{T}_{k}}^{\text {patch }}} of each template ${\mathcal{T}}_k$${\mathcal{T}}_k$T_(k)\mathcal{T}_{k}. For each proposal $m$$m$mm, we crop the detected region out from $\mathcal{I}$$\mathcal{I}$I\mathcal{I}, and resize it to a fixed resolution. The image crop is denoted as ${\mathcal{I}}_m$${\mathcal{I}}_m$I_(m)\mathcal{I}_{m} and also processed through the same ViT to obtain the class embedding ${\mathit{f}}_{{\mathcal{I}}_m}^{cls}$${\mathit{f}}_{{\mathcal{I}}_m}^{cls}$f_(I_(m))^(cls)\boldsymbol{f}_{\mathcal{I}_{m}}^{c l s} and the patch embeddings ${\left\{{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch}}\right\}}_{j=1}^{N_{{\mathcal{I}}_m}^{\text{patch}}}$${\left\{{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch }}\right\}}_{j=1}^{N_{{\mathcal{I}}_m}^{\text{patch }}}${f_(I_(m),j)^("patch ")}_(j=1)^(N_(I_(m))^("patch "))\left\{\boldsymbol{f}_{\mathcal{I}_{m}, j}^{\text {patch }}\right\}_{j=1}^{N_{\mathcal{I}_{m}}^{\text {patch }}}, with $N_{{\mathcal{I}}_m}^{\text{patch}}$$N_{{\mathcal{I}}_m}^{\text{patch }}$N_(I_(m))^("patch ")N_{\mathcal{I}_{m}}^{\text {patch }} denoting the number of patches within the object mask. Subsequently, we calculate the values of the individual score terms.
到预训练的视觉变换器（ViT）主干网络[8]，即 DINOv2[45]，从而得到每个模板 ${\mathcal{T}}_k$${\mathcal{T}}_k$T_(k)\mathcal{T}_{k} 的类别嵌入 ${\mathit{f}}_{{\mathcal{T}}_k}^{cls}$${\mathit{f}}_{{\mathcal{T}}_k}^{cls}$f_(T_(k))^(cls)\boldsymbol{f}_{\mathcal{T}_{k}}^{c l s} 和 $N_{{\mathcal{T}}_k}^{\text{patch}}$$N_{{\mathcal{T}}_k}^{\text{patch }}$N_(T_(k))^("patch ")N_{\mathcal{T}_{k}}^{\text {patch }} 个补丁嵌入 ${\left\{{\mathit{f}}_{{\mathcal{T}}_k,i}^{\text{patch}}\right\}}_{i=1}^{N_{{\mathcal{T}}_k}^{\text{patch}}}$${\left\{{\mathit{f}}_{{\mathcal{T}}_k,i}^{\text{patch }}\right\}}_{i=1}^{N_{{\mathcal{T}}_k}^{\text{patch }}}${f_(T_(k),i)^("patch ")}_(i=1)^(N_(T_(k))^("patch "))\left\{\boldsymbol{f}_{\mathcal{T}_{k}, i}^{\text {patch }}\right\}_{i=1}^{N_{\mathcal{T}_{k}}^{\text {patch }}} 。对于每个提议 $m$$m$mm ，我们从 $\mathcal{I}$$\mathcal{I}$I\mathcal{I} 中裁剪出检测到的区域，并调整到固定分辨率。该图像裁剪记为 ${\mathcal{I}}_m$${\mathcal{I}}_m$I_(m)\mathcal{I}_{m} ，同样通过相同的 ViT 处理以获得类别嵌入 ${\mathit{f}}_{{\mathcal{I}}_m}^{cls}$${\mathit{f}}_{{\mathcal{I}}_m}^{cls}$f_(I_(m))^(cls)\boldsymbol{f}_{\mathcal{I}_{m}}^{c l s} 和补丁嵌入 ${\left\{{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch}}\right\}}_{j=1}^{N_{{\mathcal{I}}_m}^{\text{patch}}}$${\left\{{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch }}\right\}}_{j=1}^{N_{{\mathcal{I}}_m}^{\text{patch }}}${f_(I_(m),j)^("patch ")}_(j=1)^(N_(I_(m))^("patch "))\left\{\boldsymbol{f}_{\mathcal{I}_{m}, j}^{\text {patch }}\right\}_{j=1}^{N_{\mathcal{I}_{m}}^{\text {patch }}} ，其中 $N_{{\mathcal{I}}_m}^{\text{patch}}$$N_{{\mathcal{I}}_m}^{\text{patch }}$N_(I_(m))^("patch ")N_{\mathcal{I}_{m}}^{\text {patch }} 表示物体掩码内的补丁数量。随后，我们计算各个得分项的数值。

Semantic Matching Score We compute a semantic score $s_{\text{sem}}$$s_{\text{sem }}$s_("sem ")s_{\text {sem }} through the class embeddings by averaging the top K values from ${\left\{\frac{⟨{\mathit{f}}_{{\mathcal{I}}_m}^{cls},{\mathit{f}}_{{\mathcal{T}}_k}^{cls}⟩}{\left|{\mathit{f}}_{{\mathcal{I}}_m}^{cls}\right|\cdot \left|{\mathit{f}}_{{\mathcal{T}}_k}^{cls}\right|}\right\}}_{k=1}^{N_{\mathcal{T}}}$${\left\{\frac{⟨{\mathit{f}}_{{\mathcal{I}}_m}^{cls},{\mathit{f}}_{{\mathcal{T}}_k}^{cls}⟩}{\left|{\mathit{f}}_{{\mathcal{I}}_m}^{cls}\right|\cdot \left|{\mathit{f}}_{{\mathcal{T}}_k}^{cls}\right|}\right\}}_{k=1}^{N_{\mathcal{T}}}${((:f_(I_(m))^(cls),f_(T_(k))^(cls):))/(|f_(I_(m))^(cls)|*|f_(T_(k))^(cls)|)}_(k=1)^(N_(T))\left\{\frac{\left\langle\boldsymbol{f}_{\mathcal{I}_{m}}^{c l s}, \boldsymbol{f}_{\mathcal{T}_{k}}^{c l s}\right\rangle}{\left|\boldsymbol{f}_{\mathcal{I}_{m}}^{c l s}\right| \cdot\left|\boldsymbol{f}_{\mathcal{T}_{k}}^{c l s}\right|}\right\}_{k=1}^{N_{\mathcal{T}}} to establish a robust measure of semantic matching, with $<,>$$<,>$< , ><,> denoting an inner product. The template that yields the highest semantic value can be seen as the best-matched template, denoted as ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }}, and is used in the computation of the subsequent two scores.
语义匹配得分 我们通过类别嵌入计算语义得分 $s_{\text{sem}}$$s_{\text{sem }}$s_("sem ")s_{\text {sem }} ，方法是从 ${\left\{\frac{⟨{\mathit{f}}_{{\mathcal{I}}_m}^{cls},{\mathit{f}}_{{\mathcal{T}}_k}^{cls}⟩}{\left|{\mathit{f}}_{{\mathcal{I}}_m}^{cls}\right|\cdot \left|{\mathit{f}}_{{\mathcal{T}}_k}^{cls}\right|}\right\}}_{k=1}^{N_{\mathcal{T}}}$${\left\{\frac{⟨{\mathit{f}}_{{\mathcal{I}}_m}^{cls},{\mathit{f}}_{{\mathcal{T}}_k}^{cls}⟩}{\left|{\mathit{f}}_{{\mathcal{I}}_m}^{cls}\right|\cdot \left|{\mathit{f}}_{{\mathcal{T}}_k}^{cls}\right|}\right\}}_{k=1}^{N_{\mathcal{T}}}${((:f_(I_(m))^(cls),f_(T_(k))^(cls):))/(|f_(I_(m))^(cls)|*|f_(T_(k))^(cls)|)}_(k=1)^(N_(T))\left\{\frac{\left\langle\boldsymbol{f}_{\mathcal{I}_{m}}^{c l s}, \boldsymbol{f}_{\mathcal{T}_{k}}^{c l s}\right\rangle}{\left|\boldsymbol{f}_{\mathcal{I}_{m}}^{c l s}\right| \cdot\left|\boldsymbol{f}_{\mathcal{T}_{k}}^{c l s}\right|}\right\}_{k=1}^{N_{\mathcal{T}}} 中取前 K 个值求平均，以建立稳健的语义匹配度量，其中 $<,>$$<,>$< , ><,> 表示内积。产生最高语义值的模板被视为最佳匹配模板，记为 ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }} ，并用于后续两个得分的计算。
Appearance Matching Score Given ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }}, we compare ${\mathcal{I}}_m$${\mathcal{I}}_m$I_(m)\mathcal{I}_{m} and ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }} in terms of appearance using an appearance score $s_{\text{appe}}$$s_{\text{appe }}$s_("appe ")s_{\text {appe }}, based on the patch embeddings, as follows:
外观匹配得分 给定 ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }} ，我们基于补丁嵌入，通过外观得分 $s_{\text{appe}}$$s_{\text{appe }}$s_("appe ")s_{\text {appe }} 比较 ${\mathcal{I}}_m$${\mathcal{I}}_m$I_(m)\mathcal{I}_{m} 和 ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }} 的外观，具体如下：
$s_{\text{appe}}=\frac{1}{N_{{\mathcal{I}}_m}^{\text{patch}}}\sum _{j=1}^{N_{{\mathcal{I}}_m}^{\text{patch}}}\underset{i=1,\dots ,N_{{\mathcal{T}}_{\text{best}}}^{\text{patch}}}{max}\frac{⟨{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch}},{\mathit{f}}_{{\mathcal{T}}_{\text{best}},i}^{\text{patch}}⟩}{\left|{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch}}\right|\cdot \left|{\mathit{f}}_{{\mathcal{T}}_{\text{best}},i}^{\text{patch}}\right|}$$s_{\text{appe }}=\frac{1}{N_{{\mathcal{I}}_m}^{\text{patch }}}\sum _{j=1}^{N_{{\mathcal{I}}_m}^{\text{patch }}} \underset{i=1,\dots ,N_{{\mathcal{T}}_{\text{best }}}^{\text{patch }}}{max} \frac{⟨{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch }},{\mathit{f}}_{{\mathcal{T}}_{\text{best }},i}^{\text{patch }}⟩}{\left|{\mathit{f}}_{{\mathcal{I}}_m,j}^{\text{patch }}\right|\cdot \left|{\mathit{f}}_{{\mathcal{T}}_{\text{best }},i}^{\text{patch }}\right|}$s_("appe ")=(1)/(N_(I_(m))^("patch "))sum_(j=1)^(N_(I_(m))^("patch "))max_(i=1,dots,N_(T_("best "))^("patch "))((:f_(I_(m),j)^("patch "),f_(T_("best "),i)^("patch "):))/(|f_(I_(m),j)^("patch ")|*|f_(T_("best "),i)^("patch ")|)s_{\text {appe }}=\frac{1}{N_{\mathcal{I}_{m}}^{\text {patch }}} \sum_{j=1}^{N_{\mathcal{I}_{m}}^{\text {patch }}} \max _{i=1, \ldots, N_{\mathcal{T}_{\text {best }}}^{\text {patch }}} \frac{\left\langle\boldsymbol{f}_{\mathcal{I}_{m}, j}^{\text {patch }}, \boldsymbol{f}_{\mathcal{T}_{\text {best }}, i}^{\text {patch }}\right\rangle}{\left|\boldsymbol{f}_{\mathcal{I}_{m}, j}^{\text {patch }}\right| \cdot\left|\boldsymbol{f}_{\mathcal{T}_{\text {best }}, i}^{\text {patch }}\right|}.
$s_{\text{appe}}$$s_{\text{appe }}$s_("appe ")s_{\text {appe }} is utilized to distinguish objects that are semantically similar but differ in appearance.
$s_{\text{appe}}$$s_{\text{appe }}$s_("appe ")s_{\text {appe }} 用于区分语义相似但外观不同的物体。
Geometric Matching Score In terms of geometry, we score the proposal $m$$m$mm by considering factors like object shapes and sizes. Utilizing the object rotation from ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }} and the mean location of the cropped points of $m$$m$mm, we have a coarse pose to transform the object $\mathcal{O}$$\mathcal{O}$O\mathcal{O}, which is then projected onto the image to obtain a compact bounding box ${\mathcal{B}}_o$${\mathcal{B}}_o$B_(o)\mathcal{B}_{o}. Afterwards, the Intersection-over-Union (IoU) value between ${\mathcal{B}}_o$${\mathcal{B}}_o$B_(o)\mathcal{B}_{o} and the bounding box ${\mathcal{B}}_m$${\mathcal{B}}_m$B_(m)\mathcal{B}_{m} of $m$$m$mm is used as the geometric score $s_{\text{geo}}$$s_{\text{geo }}$s_("geo ")s_{\text {geo }} :
几何匹配得分 在几何方面，我们通过考虑物体的形状和大小等因素来对提议 $m$$m$mm 进行评分。利用来自 ${\mathcal{T}}_{\text{best}}$${\mathcal{T}}_{\text{best }}$T_("best ")\mathcal{T}_{\text {best }} 的物体旋转和 $m$$m$mm 裁剪点的平均位置，我们得到了一个粗略的姿态，用于变换物体 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} ，然后将其投影到图像上以获得一个紧凑的边界框 ${\mathcal{B}}_o$${\mathcal{B}}_o$B_(o)\mathcal{B}_{o} 。随后，使用 ${\mathcal{B}}_o$${\mathcal{B}}_o$B_(o)\mathcal{B}_{o} 与 $m$$m$mm 的边界框 ${\mathcal{B}}_m$${\mathcal{B}}_m$B_(m)\mathcal{B}_{m} 之间的交并比（IoU）值作为几何得分 $s_{\text{geo}}$$s_{\text{geo }}$s_("geo ")s_{\text {geo }} ：

The reliability of $s_{\text{geo}}$$s_{\text{geo }}$s_("geo ")s_{\text {geo }} is easily impacted by occlusions. We thus compute a visible ratio $r_{vis}$$r_{vis}$r_(vis)r_{v i s} to evaluate the confidence of $s_{\text{geo}}$$s_{\text{geo }}$s_("geo ")s_{\text {geo }}, which is detailed in the supplementary materials.
$s_{\text{geo}}$$s_{\text{geo }}$s_("geo ")s_{\text {geo }} 的可靠性容易受到遮挡的影响。因此，我们计算了一个可见比例 $r_{vis}$$r_{vis}$r_(vis)r_{v i s} 来评估 $s_{\text{geo}}$$s_{\text{geo }}$s_("geo ")s_{\text {geo }} 的置信度，具体内容详见补充材料。

By combining the above three score terms, the object matching score $s_m$$s_m$s_(m)s_{m} could be formulated as follows:
通过结合上述三个得分项，物体匹配得分 $s_m$$s_m$s_(m)s_{m} 可表示为：

SAM-6D uses a Pose Estimation Model (PEM) to predict the 6D poses of the proposals matched with the object $\mathcal{O}$$\mathcal{O}$O\mathcal{O}.
SAM-6D 使用姿态估计模型（PEM）来预测与物体 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 匹配的提议的 6D 姿态。

For each object proposal $m$$m$mm, PEM uses a strategy of point registration to predict the 6 D pose w.r.t. $\mathcal{O}$$\mathcal{O}$O\mathcal{O}. Denoting the sampled point set of $m$$m$mm as ${\mathcal{P}}_m\in {\mathbb{R}}^{N_m\times 3}$${\mathcal{P}}_m\in {\mathbb{R}}^{N_m\times 3}$P_(m)inR^(N_(m)xx3)\mathcal{P}_{m} \in \mathbb{R}^{N_{m} \times 3} with $N_m$$N_m$N_(m)N_{m} points
对于每个物体提议 $m$$m$mm ，PEM 使用点注册策略来预测相对于 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 的 6D 姿态。将 $m$$m$mm 的采样点集表示为 ${\mathcal{P}}_m\in {\mathbb{R}}^{N_m\times 3}$${\mathcal{P}}_m\in {\mathbb{R}}^{N_m\times 3}$P_(m)inR^(N_(m)xx3)\mathcal{P}_{m} \in \mathbb{R}^{N_{m} \times 3} ，包含 $N_m$$N_m$N_(m)N_{m} 个点

Figure 3. An illustration of Pose Estimation Model (PEM) of SAM-6D.
图 3. SAM-6D 姿态估计模型（PEM）示意图。
and that of $\mathcal{O}$$\mathcal{O}$O\mathcal{O} as ${\mathcal{P}}_o\in {\mathbb{R}}^{N_o\times 3}$${\mathcal{P}}_o\in {\mathbb{R}}^{N_o\times 3}$P_(o)inR^(N_(o)xx3)\mathcal{P}_{o} \in \mathbb{R}^{N_{o} \times 3} with $N_o$$N_o$N_(o)N_{o} points, the goal is to solve an assignment matrix to present the partial-to-partial correspondence between ${\mathcal{P}}_m$${\mathcal{P}}_m$P_(m)\mathcal{P}_{m} and ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o}. Partial-to-partial correspondence arises as ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} only partially matches ${\mathcal{P}}_m$${\mathcal{P}}_m$P_(m)\mathcal{P}_{m} due to occlusions, and ${\mathcal{P}}_m$${\mathcal{P}}_m$P_(m)\mathcal{P}_{m} may partially align with ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} due to segmentation inaccuracies and sensor noises. We propose to equip their respective point features ${\mathit{F}}_m\in {\mathbb{R}}^{N_m\times C}$${\mathit{F}}_m\in {\mathbb{R}}^{N_m\times C}$F_(m)inR^(N_(m)xx C)\boldsymbol{F}_{m} \in \mathbb{R}^{N_{m} \times C} and ${\mathit{F}}_o\in {\mathbb{R}}^{N_o\times C}$${\mathit{F}}_o\in {\mathbb{R}}^{N_o\times C}$F_(o)inR^(N_(o)xx C)\boldsymbol{F}_{o} \in \mathbb{R}^{N_{o} \times C} with learnable Background Tokens, denoted as ${\mathit{f}}_m^{bg}\in {\mathbb{R}}^C$${\mathit{f}}_m^{bg}\in {\mathbb{R}}^C$f_(m)^(bg)inR^(C)\boldsymbol{f}_{m}^{b g} \in \mathbb{R}^{C} and ${\mathit{f}}_o^{bg}\in {\mathbb{R}}^C$${\mathit{f}}_o^{bg}\in {\mathbb{R}}^C$f_(o)^(bg)inR^(C)\boldsymbol{f}_{o}^{b g} \in \mathbb{R}^{C}, where $C$$C$CC is the number of feature channels. This simple design resolves the assignment problem of non-overlapped points in two point sets, and the partial-to-partial correspondence thus could be effectively built based on feature similarities. Specifically, we can first compute the attention matrix $\mathcal{A}$$\mathcal{A}$A\mathcal{A} as follows:
以及 $\mathcal{O}$$\mathcal{O}$O\mathcal{O} 作为 ${\mathcal{P}}_o\in {\mathbb{R}}^{N_o\times 3}$${\mathcal{P}}_o\in {\mathbb{R}}^{N_o\times 3}$P_(o)inR^(N_(o)xx3)\mathcal{P}_{o} \in \mathbb{R}^{N_{o} \times 3} ，具有 $N_o$$N_o$N_(o)N_{o} 个点，目标是求解一个分配矩阵，以表示 ${\mathcal{P}}_m$${\mathcal{P}}_m$P_(m)\mathcal{P}_{m} 和 ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} 之间的部分到部分对应关系。部分到部分对应关系的产生是因为 ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} 仅部分匹配 ${\mathcal{P}}_m$${\mathcal{P}}_m$P_(m)\mathcal{P}_{m} ，这是由于遮挡造成的，而 ${\mathcal{P}}_m$${\mathcal{P}}_m$P_(m)\mathcal{P}_{m} 可能由于分割不准确和传感器噪声而部分与 ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} 对齐。我们提出为它们各自的点特征 ${\mathit{F}}_m\in {\mathbb{R}}^{N_m\times C}$${\mathit{F}}_m\in {\mathbb{R}}^{N_m\times C}$F_(m)inR^(N_(m)xx C)\boldsymbol{F}_{m} \in \mathbb{R}^{N_{m} \times C} 和 ${\mathit{F}}_o\in {\mathbb{R}}^{N_o\times C}$${\mathit{F}}_o\in {\mathbb{R}}^{N_o\times C}$F_(o)inR^(N_(o)xx C)\boldsymbol{F}_{o} \in \mathbb{R}^{N_{o} \times C} 配备可学习的背景标记，分别记为 ${\mathit{f}}_m^{bg}\in {\mathbb{R}}^C$${\mathit{f}}_m^{bg}\in {\mathbb{R}}^C$f_(m)^(bg)inR^(C)\boldsymbol{f}_{m}^{b g} \in \mathbb{R}^{C} 和 ${\mathit{f}}_o^{bg}\in {\mathbb{R}}^C$${\mathit{f}}_o^{bg}\in {\mathbb{R}}^C$f_(o)^(bg)inR^(C)\boldsymbol{f}_{o}^{b g} \in \mathbb{R}^{C} ，其中 $C$$C$CC 是特征通道的数量。这一简单设计解决了两个点集之间非重叠点的分配问题，因此部分到部分对应关系可以基于特征相似性有效建立。具体来说，我们可以首先计算注意力矩阵 $\mathcal{A}$$\mathcal{A}$A\mathcal{A} ，计算方式如下：

and then obtain the soft assignment matrix $\tilde{\mathcal{A}}$$\tilde{\mathcal{A}}$tilde(A)\tilde{\mathcal{A}}
然后得到软分配矩阵 $\tilde{\mathcal{A}}$$\tilde{\mathcal{A}}$tilde(A)\tilde{\mathcal{A}}

where ${\mathrm{Softmax}}_{\text{row}}()$${\mathrm{Softmax}}_{\text{row }}()$Softmax_("row ")()\operatorname{Softmax}_{\text {row }}() and ${\mathrm{Softmax}}_{\text{col}}()$${\mathrm{Softmax}}_{\text{col }}()$Softmax_("col ")()\operatorname{Softmax}_{\text {col }}() denote Softmax operations executed along the row and column of the matrix, respectively. $\tau$$\tau$tau\tau is a constant temperature. The values in each row of $\tilde{\mathcal{A}}$$\tilde{\mathcal{A}}$tilde(A)\tilde{\mathcal{A}}, excluding the first row associated with the background, indicate the matching probabilities of the point ${\mathit{p}}_m\in {\mathcal{P}}_m$${\mathit{p}}_m\in {\mathcal{P}}_m$p_(m)inP_(m)\boldsymbol{p}_{m} \in \mathcal{P}_{m} aligning with background and the points in ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o}. Specifically, for ${\mathit{p}}_m$${\mathit{p}}_m$p_(m)\boldsymbol{p}_{m}, its corresponding point ${\mathit{p}}_o\in {\mathcal{P}}_o$${\mathit{p}}_o\in {\mathcal{P}}_o$p_(o)inP_(o)\boldsymbol{p}_{o} \in \mathcal{P}_{o} can be identified by locating the index of the maximum score $\tilde{a}\in \tilde{\mathcal{A}}$$\tilde{a}\in \tilde{\mathcal{A}}$tilde(a)in tilde(A)\tilde{a} \in \tilde{\mathcal{A}} along the row; if this index equals zero, the embedding of ${\mathit{p}}_m$${\mathit{p}}_m$p_(m)\boldsymbol{p}_{m} aligns with the background token, indicating it
其中 ${\mathrm{Softmax}}_{\text{row}}()$${\mathrm{Softmax}}_{\text{row }}()$Softmax_("row ")()\operatorname{Softmax}_{\text {row }}() 和 ${\mathrm{Softmax}}_{\text{col}}()$${\mathrm{Softmax}}_{\text{col }}()$Softmax_("col ")()\operatorname{Softmax}_{\text {col }}() 分别表示沿矩阵的行和列执行的 Softmax 操作。 $\tau$$\tau$tau\tau 是一个常数温度。 $\tilde{\mathcal{A}}$$\tilde{\mathcal{A}}$tilde(A)\tilde{\mathcal{A}} 中每一行的值，除与背景相关的第一行外，表示点 ${\mathit{p}}_m\in {\mathcal{P}}_m$${\mathit{p}}_m\in {\mathcal{P}}_m$p_(m)inP_(m)\boldsymbol{p}_{m} \in \mathcal{P}_{m} 与背景及 ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} 中点的匹配概率。具体来说，对于 ${\mathit{p}}_m$${\mathit{p}}_m$p_(m)\boldsymbol{p}_{m} ，其对应的点 ${\mathit{p}}_o\in {\mathcal{P}}_o$${\mathit{p}}_o\in {\mathcal{P}}_o$p_(o)inP_(o)\boldsymbol{p}_{o} \in \mathcal{P}_{o} 可以通过沿行定位最大分数 $\tilde{a}\in \tilde{\mathcal{A}}$$\tilde{a}\in \tilde{\mathcal{A}}$tilde(a)in tilde(A)\tilde{a} \in \tilde{\mathcal{A}} 的索引来确定；如果该索引等于零，则表示 ${\mathit{p}}_m$${\mathit{p}}_m$p_(m)\boldsymbol{p}_{m} 的嵌入与背景标记对齐，表明它
has no valid correspondence in ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o}. Once $\tilde{\mathcal{A}}$$\tilde{\mathcal{A}}$tilde(A)\tilde{\mathcal{A}} is obtained, we can gather all the matched pairs $\left\{({\mathit{p}}_m,{\mathit{p}}_o)\right\}$$\left\{\left({\mathit{p}}_m,{\mathit{p}}_o\right)\right\}${(p_(m),p_(o))}\left\{\left(\boldsymbol{p}_{m}, \boldsymbol{p}_{o}\right)\right\}, along with their scores $\{\tilde{a}\}$$\{\tilde{a}\}${ tilde(a)}\{\tilde{a}\}, to compute the pose using weighted SVD.
在 ${\mathcal{P}}_o$${\mathcal{P}}_o$P_(o)\mathcal{P}_{o} 中没有有效对应。一旦获得 $\tilde{\mathcal{A}}$$\tilde{\mathcal{A}}$tilde(A)\tilde{\mathcal{A}} ，我们可以收集所有匹配对 $\left\{({\mathit{p}}_m,{\mathit{p}}_o)\right\}$$\left\{\left({\mathit{p}}_m,{\mathit{p}}_o\right)\right\}${(p_(m),p_(o))}\left\{\left(\boldsymbol{p}_{m}, \boldsymbol{p}_{o}\right)\right\} 及其分数 $\{\tilde{a}\}$$\{\tilde{a}\}${ tilde(a)}\{\tilde{a}\} ，利用加权 SVD 计算姿态。