Query-efficient Attack for Black-box Image Inpainting Forensics via Reinforcement Learning
基于强化学习的查询高效攻击方法在黑盒图像修复取证中的应用

The rapid development of image forgery technologies such as image inpainting has seriously challenged the authenticity and integrity of digital images (Verdoliva 2020). As digital images are frequently employed as evidence and reliable records in contexts such as criminal investigations, journalistic reporting, and intellectual property protection, the detection and localization of inpainted regions within images have emerged as significant research challenges. Numerous forensic methods(Mayer and Stamm 2018; Li and Huang 2019; Wu and Zhou 2022; Yang, Cai, and Kot 2022; Zhang et al. 2023) based on deep learning have been proposed to address these issues in the past decade.
图像生成技术（如图像修复）的快速发展对数字图像的真实性和完整性构成了严峻挑战（Verdoliva 2020）。由于数字图像常被用作刑事调查、新闻报道和知识产权保护等场景中的证据和可靠记录，检测并定位图像中的修复区域已成为重要的研究难题。过去十年间，基于深度学习的多种法医方法（Mayer and Stamm 2018；Li and Huang 2019；Wu and Zhou 2022；Yang, Cai, and Kot 2022；Zhang et al. 2023）已被提出以解决这些问题。

Although deep learning technology has significantly advanced inpainting forensics, current research (Akhtar and
尽管深度学习技术在图像修复取证领域取得了显著进展，但当前研究（Akhtar 和

Mian 2018) has shown that it is vulnerable to adversarial attacks. These attacks introduce perturbations to manipulate input data, thereby misleading the target network into producing incorrect predictions. They are typically classified as either white-box or black-box attacks, depending on the level of prior knowledge about the target network. Whitebox attacks have full access to the details of the target network, while black-box attacks have no access to these internal details. In inpainting forensics, adversarial attacks pose a severe challenge. The reason is that inpainting forensics methods remain the only reliable way to identify inpainting regions, as human eyes struggle to detect manipulated areas within meticulously crafted inpainting images. Consequently, the incorrect forensics results caused by adversarial attacks are of utmost concern. Moreover, since anti-forensics attacks expose vulnerabilities in existing forensics methods, research on these attacks can further promote the design of more robust forensics methods. To address these challenges, many anti-forensics methods have been proposed in recent years (Barni et al. 2019; Carlini and Farid 2020; Xie, Ni, and Shi 2021; Ding et al. 2022; Fan, Hu, and Ding 2024).
Mian（2018）的研究表明，该方法易受对抗攻击的影响。此类攻击通过引入扰动来操纵输入数据，从而误导目标网络产生错误预测。根据对目标网络的先验知识程度，对抗攻击通常被分为白盒攻击和黑盒攻击。白盒攻击能够完全访问目标网络的内部细节，而黑盒攻击则无法获取这些内部信息。在图像修复取证领域，对抗性攻击构成重大挑战。原因在于，填充取证方法仍是识别填充区域的唯一可靠手段，因为人眼难以在精心制作的填充图像中检测出被篡改的区域。因此，对抗攻击导致的错误取证结果尤为令人担忧。此外，由于反取证攻击会暴露现有取证方法的漏洞，对这些攻击的研究可进一步推动更 robust 取证方法的设计。为应对这些挑战，近年来提出了多种反取证方法（Barni 等，2019；Carlini 和 Farid，2020；Xie、Ni 和 Shi，2021；Ding 等，2022；Fan、Hu 和 Ding，2024）。

However, existing attack methods are inappropriate for image inpainting forensics. First, current anti-forensics methods have focused on attacking image-wise classifiers, whereas pixel-wise segmentation networks are used in inpainting forensics. Secondly, in real-world scenarios, inpainting forensics operate as black-box systems, with only query results available to attackers. This makes querybased attacks the most suitable approach for inpainting forensics. Unfortunately, existing anti-forensics methods are predominantly designed for white-box attacks, with several relying on transfer-based black-box attacks. Besides, for query-based black-box adversarial examples methods $(\mathrm{Li}$$(\mathrm{Li}$(Li(\mathrm{Li} and Chen 2021; Andriushchenko et al. 2020; Maho, Furon, and Le Merrer 2021) designed for computer vision tasks, current research(Li et al. 2022) has shown that they can be easily detected based on the similarity of successive queries. Nevertheless, this detection method suffers from extremely high false alarm rates when the number of queries is less than 10 . Therefore, there is an urgent need for queryefficient anti-forensics methods tailored for black-box image inpainting forensics, capable of constructing successful attacks with fewer than 10 queries.
然而，现有的攻击方法并不适用于图像修复取证分析。首先，当前的反取证方法主要针对图像级分类器进行攻击，而图像修复取证分析则采用像素级分割网络。其次，在实际场景中，图像修复取证作为黑盒系统运行，攻击者仅能获取查询结果。这使得基于查询的攻击成为图像修复取证中最合适的方法。遗憾的是，现有反取证方法主要针对白盒攻击设计，其中部分方法依赖于基于迁移的黑盒攻击。此外，针对计算机视觉任务设计的基于查询的黑盒对抗样本方法（如 $(\mathrm{Li}$$(\mathrm{Li}$(Li(\mathrm{Li} 和 Chen 2021；Andriushchenko et al. 2020；Maho, Furon, and Le Merrer 2021），当前研究（Li et al. 2022）表明，这些方法可通过连续查询的相似性被轻松检测。然而，当查询数量少于 10 时，该检测方法存在极高的误报率。因此，迫切需要针对黑盒图像修复取证的查询高效反取证方法，能够使用少于 10 个查询构建成功的攻击。

To address this, we propose RLGC(Reinforcement
为了解决这个问题，我们提出 RLGC（强化学习）

Learning to Generate Countermeasure) to conduct highly query-efficient attack for black-box image inpainting forensics methods. Our goal is to utilize query results to build adversarial attacks and achieve minimum visual distortion on the original inpainting images. Specifically, we limit the query times of RLGC to less than 10 times to evade the detection of query-based attack defense method(Li et al. 2022). To this end, we first model attack scenario based on RL(Reinforcement Learning) paradigm. Given the original inpainting images, which correspond to the initial state, the agent selects actions based on its policy and conducts state transitions to modulate adversarial perturbations for anti-forensics images. These images are then used to query target forensics networks, obtaining corresponding outputs. The reward function evaluates the attack effect and visual distortion according to these outputs. To maximize the cumulative reward, our policy and value network are integrated by Asynchronous Advantage Actor-Critic (A3C) framework, where the advantage-based loss functions optimize network parameters. Through iterative interactions between agents and image inpainting forensics methods, RLGC’s attack efficiency can be constantly optimized until it achieves the given goal. Our contributions are as follows: (1) We propose the first query-based anti-forensics framework targeting black-box inpainting forensic methods, thereby eliminating the dependence on the transferability of white-box attacks. (2) We first apply a reinforcement learning (RL) paradigm within the anti-forensics framework, enabling pixel-wise agents to learn highly query-efficient policies based on inpainting forensics results. (3) We propose a novel method for generating perturbations that incrementally introduce small magnitudes( $+1/-1/0$$+1/-1/0$+1//-1//0+1 /-1 / 0 ) of noise, thus mitigating the risk of generating excessively strong noise that could leave conspicuous attack traces.
学习生成反制措施，以对黑盒图像修复取证方法进行高查询效率的攻击。我们的目标是利用查询结果构建对抗性攻击，并在原始修复图像上实现最小视觉失真。具体而言，我们将 RLGC 的查询次数限制在 10 次以内，以规避基于查询的攻击防御方法（Li et al. 2022）的检测。为此，我们首先基于强化学习（RL）范式建模攻击场景。给定原始修复图像（对应初始状态），代理根据策略选择动作并进行状态转换，以生成对抗性扰动用于生成反取证图像。这些图像随后用于查询目标取证网络，获取对应输出。奖励函数根据这些输出评估攻击效果和视觉失真。为了最大化累积奖励，我们的策略和价值网络通过异步优势演员-批评家（A3C）框架进行集成，其中基于优势的损失函数优化网络参数。通过代理与图像修复取证方法的迭代交互，RLGC 的攻击效率可不断优化直至达到给定目标。我们的贡献如下：(1) 我们提出了首个针对黑盒图像修复取证方法的查询式反取证框架，从而消除了对白盒攻击可迁移性的依赖。(2) 我们首次在反取证框架中应用强化学习（RL）范式，使像素级代理能够基于图像修复取证结果学习高度查询高效的策略。 (3) 我们提出了一种新型方法，用于生成逐步引入小幅度（ $+1/-1/0$$+1/-1/0$+1//-1//0+1 /-1 / 0 ）噪声的扰动，从而降低生成过强噪声的风险，避免留下明显的攻击痕迹。

Given a mask $\{\mathbf{M}={\left(m_{i,j}\right)}^{(w\times h)},m_{i,j}\in \{0,1\}\}$$\left\{\mathbf{M}={\left(m_{i,j}\right)}^{(w\times h)},m_{i,j}\in \{0,1\}\right\}${M=(m_(i,j))^((w xx h)),m_(i,j)in{0,1}}\left\{\mathbf{M}=\left(m_{i, j}\right)^{(w \times h)}, m_{i, j} \in\{0,1\}\right\}, the damaged image can be calculated as: $\mathbf{D}={\left(d_{i,j,k}\right)}^{(w\times h\times c)}=$$\mathbf{D}={\left(d_{i,j,k}\right)}^{(w\times h\times c)}=$D=(d_(i,j,k))^((w xx h xx c))=\mathbf{D}=\left(d_{i, j, k}\right)^{(w \times h \times c)}= ${(x_{i,j,k}\ast m_{i,j})}^{(w\times h\times c)}$${\left(x_{i,j,k}\ast m_{i,j}\right)}^{(w\times h\times c)}$(x_(i,j,k)**m_(i,j))^((w xx h xx c))\left(x_{i, j, k} * m_{i, j}\right)^{(w \times h \times c)}, image inpainting is to obtain an inpainting image $\mathbf{Y}$$\mathbf{Y}$Y\mathbf{Y} that satisfies:
给定一个掩码 $\{\mathbf{M}={\left(m_{i,j}\right)}^{(w\times h)},m_{i,j}\in \{0,1\}\}$$\left\{\mathbf{M}={\left(m_{i,j}\right)}^{(w\times h)},m_{i,j}\in \{0,1\}\right\}${M=(m_(i,j))^((w xx h)),m_(i,j)in{0,1}}\left\{\mathbf{M}=\left(m_{i, j}\right)^{(w \times h)}, m_{i, j} \in\{0,1\}\right\} ，损坏的图像可以计算为： $\mathbf{D}={\left(d_{i,j,k}\right)}^{(w\times h\times c)}=$$\mathbf{D}={\left(d_{i,j,k}\right)}^{(w\times h\times c)}=$D=(d_(i,j,k))^((w xx h xx c))=\mathbf{D}=\left(d_{i, j, k}\right)^{(w \times h \times c)}= ${(x_{i,j,k}\ast m_{i,j})}^{(w\times h\times c)}$${\left(x_{i,j,k}\ast m_{i,j}\right)}^{(w\times h\times c)}$(x_(i,j,k)**m_(i,j))^((w xx h xx c))\left(x_{i, j, k} * m_{i, j}\right)^{(w \times h \times c)} ，图像修复的目标是获得一个修复图像 $\mathbf{Y}$$\mathbf{Y}$Y\mathbf{Y} ，满足以下条件：

where $\Vert \cdot \Vert$$\Vert \cdot \Vert$||*||\|\cdot\| is L 2 norm, and ${\theta }_i$${\theta }_i$theta_(i)\theta_{i} is the inpainting algorithm.
其中 $\Vert \cdot \Vert$$\Vert \cdot \Vert$||*||\|\cdot\| 表示 L² 范数， ${\theta }_i$${\theta }_i$theta_(i)\theta_{i} 表示插值算法。
Over the years, advancements in inpainting frameworks have resulted in synthetic images that are increasingly difficult to distinguish from authentic ones. Consequently, various forensics methods have been proposed for detecting image inpainting (Mayer and Stamm 2018; Li and Huang 2019; Wu and Zhou 2022; Yang, Cai, and Kot 2022; Zhang et al. 2023). These methods not only determine the authenticity of an image but also locate its synthetic regions. Given a ground truth mask $\mathit{M}$$\mathit{M}$M\boldsymbol{M}, the objective of inpainting forensics methods ${\theta }_f$${\theta }_f$theta_(f)\theta_{f} is to predict the mask ${\mathit{M}}^{\mathit{p}}$${\mathit{M}}^{\mathit{p}}$M^(p)\boldsymbol{M}^{\boldsymbol{p}} while adhering to the following constraints:
近年来，图像修复框架的不断发展使得生成的合成图像越来越难以与真实图像区分开来。因此，研究人员提出了多种图像修复检测方法（Mayer 和 Stamm 2018；Li 和 Huang 2019；Wu 和 Zhou 2022；Yang、Cai 和 Kot 2022；Zhang 等 2023）。这些方法不仅能够判断图像的真实性，还能定位其合成区域。给定一个真实掩码 $\mathit{M}$$\mathit{M}$M\boldsymbol{M} ，图像修复取证方法 ${\theta }_f$${\theta }_f$theta_(f)\theta_{f} 的目标是在满足以下约束条件的同时预测掩码 ${\mathit{M}}^{\mathit{p}}$${\mathit{M}}^{\mathit{p}}$M^(p)\boldsymbol{M}^{\boldsymbol{p}} ：

Black-box attacks can be categorized into two types: (1) Transfer-based attacks: A local substitute model is first trained. Adversarial examples are then generated using white-box attacks such as FGSM(Goodfellow, Shlens, and Szegedy 2015) and i-FGSM(Kurakin, Goodfellow, and Bengio 2017) on the substitute model, which are later directly used to attack target models. The success of these attacks depends on the transferability of white-box adversarial examples, which is influenced by the discrepancy between target and substitute models. (2) Query-based attacks: They can be classified into two categories: score-based attacks( Li and Chen 2021; Andriushchenko et al. 2020) and decision-based attacks(Maho, Furon, and Le Merrer 2021). Score-based attacks involve adding slight perturbations to the input and observing the response of target models. In contrast, decisionbased attacks start from a point already in the adversarial region and use binary search to find a point on the decision boundary between the starting point and the clean example.
黑盒攻击可分为两类：(1) 转移式攻击：首先训练一个本地替换模型。随后，利用白盒攻击（如 FGSM（Goodfellow、Shlens 和 Szegedy，2015）和 i-FGSM（Kurakin、Goodfellow 和 Bengio，2017））在替换模型上生成对抗样本，这些样本随后直接用于攻击目标模型。这些攻击的成功取决于白盒对抗样本的迁移性，而迁移性受目标模型与替换模型之间差异的影响。(2) 查询式攻击：可分为两类：基于评分攻击（Li 和 Chen 2021；Andriushchenko 等 2020）和基于决策攻击（Maho、Furon 和 Le Merrer 2021）。基于评分攻击通过在输入中添加微小扰动并观察目标模型的响应来实现。而基于决策的攻击则从已处于对抗区域的点出发，利用二分搜索在起点与干净示例之间的决策边界上寻找一个点。

RL is one of the three fundamental machine learning paradigms, which focuses on creating intelligent agents that can take actions to maximize cumulative reward. To model a real-world scenario, a tuple ( $\mathcal{S},\mathcal{A},\pi ,r,\delta$$\mathcal{S},\mathcal{A},\pi ,r,\delta$S,A,pi,r,delta\mathcal{S}, \mathcal{A}, \pi, r, \delta ) is defined based on the existing background knowledge, where $\mathcal{S}$$\mathcal{S}$S\mathcal{S} denotes the set of states or the environment, $\mathcal{A}$$\mathcal{A}$A\mathcal{A} represents the action set, $\pi$$\pi$pi\pi reflects the probability of state transition, $r$$r$rr signifies the reward generated by the state transition, and $\delta$$\delta$delta\delta is the discount factor for rewards. Typically, practical RL algorithms are based on infinite-horizon MDP (Markov Decision Process) with successive state transitions. In a single state transition, the process starts from the current state $s_c\in \mathcal{S}$$s_c\in \mathcal{S}$s_(c)inSs_{c} \in \mathcal{S}. Then, an action $a_c\in \mathcal{A}$$a_c\in \mathcal{A}$a_(c)inAa_{c} \in \mathcal{A} is selected according to $\pi (a_c\mid s_c)$$\pi \left(a_c\mid s_c\right)$pi(a_(c)∣s_(c))\pi\left(a_{c} \mid s_{c}\right). As a result, $s_c$$s_c$s_(c)s_{c} transitions to next state $s_n\in \mathcal{S}$$s_n\in \mathcal{S}$s_(n)inSs_{n} \in \mathcal{S}, which leads to the reward $r(s_c,s_n)$$r\left(s_c,s_n\right)$r(s_(c),s_(n))r\left(s_{c}, s_{n}\right). In recent years, with the rapid development of deep learning technology, RL has been integrated with deep learning to form the deep RL paradigm.
强化学习（RL）是机器学习三大基本范式之一，其核心在于创建能够通过采取行动来最大化累计奖励的智能代理。为了建模真实世界场景，基于现有背景知识定义了一个元组（ $\mathcal{S},\mathcal{A},\pi ,r,\delta$$\mathcal{S},\mathcal{A},\pi ,r,\delta$S,A,pi,r,delta\mathcal{S}, \mathcal{A}, \pi, r, \delta ），其中 $\mathcal{S}$$\mathcal{S}$S\mathcal{S} 表示状态集或环境， $\mathcal{A}$$\mathcal{A}$A\mathcal{A} 表示动作集， $\pi$$\pi$pi\pi 反映状态转换的概率， $r$$r$rr 表示状态转换产生的奖励， $\delta$$\delta$delta\delta 是奖励的折扣因子。通常，实际的 RL 算法基于具有连续状态转换的无限 horizon MDP（马尔可夫决策过程）。在单次状态转换中，过程从当前状态 $s_c\in \mathcal{S}$$s_c\in \mathcal{S}$s_(c)inSs_{c} \in \mathcal{S} 开始。随后，根据 $\pi (a_c\mid s_c)$$\pi \left(a_c\mid s_c\right)$pi(a_(c)∣s_(c))\pi\left(a_{c} \mid s_{c}\right) 选择一个动作 $a_c\in \mathcal{A}$$a_c\in \mathcal{A}$a_(c)inAa_{c} \in \mathcal{A} 。结果， $s_c$$s_c$s_(c)s_{c} 过渡到下一个状态 $s_n\in \mathcal{S}$$s_n\in \mathcal{S}$s_(n)inSs_{n} \in \mathcal{S} ，从而获得奖励 $r(s_c,s_n)$$r\left(s_c,s_n\right)$r(s_(c),s_(n))r\left(s_{c}, s_{n}\right) 。近年来，随着深度学习技术的快速发展，RL 与深度学习相结合，形成了深度 RL 范式。

A3C(Mnih et al. 2016) is a deep RL-based algorithm. The foundation of A 3 C is an actor-critic framework, where the actor selects its actions for the current state $s_c$$s_c$s_(c)s_{c} based on $\pi (a_c\mid s_c)$$\pi \left(a_c\mid s_c\right)$pi(a_(c)∣s_(c))\pi\left(a_{c} \mid s_{c}\right), while the critic evaluates the value of the next state $s_n$$s_n$s_(n)s_{n}. Typically, deep learning-based policy and value networks are used as the actor and critic in A3C. To train these networks, A3C leverages the advantage of the actor over the critic, which is the difference between the expected reward and value. We denote the policy network and value network as $P$$P$PP and $V$$V$VV respectively, and represent their parameters as ${\theta }_p$${\theta }_p$theta_(p)\theta_{p} and ${\theta }_v$${\theta }_v$theta_(v)\theta_{v}. At time step $t$$t$tt, the expected reward of $N$$N$NN following states $\{s_{(t+i)}\mid i=0,1,\dots ,N-1\}$$\left\{s_{(t+i)}\mid i=0,1,\dots ,N-1\right\}${s_((t+i))∣i=0,1,dots,N-1}\left\{s_{(t+i)} \mid i=0,1, \ldots, N-1\right\} is calculated as:
A3C（Mnih 等，2016）是一种基于深度强化学习的算法。A3C 的基础是一个演员-批评家框架，其中演员根据当前状态 $s_c$$s_c$s_(c)s_{c} 选择其动作，而批评家评估下一个状态的价值 $s_n$$s_n$s_(n)s_{n} 。通常，深度学习基政策网络和价值网络被用作 A3C 中的演员和批评者。为了训练这些网络，A3C 利用了演员相对于批评者的优势，即预期奖励与价值之间的差异。我们分别用 $P$$P$PP 和 $V$$V$VV 表示策略网络和价值网络，并用 ${\theta }_p$${\theta }_p$theta_(p)\theta_{p} 和 ${\theta }_v$${\theta }_v$theta_(v)\theta_{v} 表示其参数。在时间步 $t$$t$tt ，状态 $\{s_{(t+i)}\mid i=0,1,\dots ,N-1\}$$\left\{s_{(t+i)}\mid i=0,1,\dots ,N-1\right\}${s_((t+i))∣i=0,1,dots,N-1}\left\{s_{(t+i)} \mid i=0,1, \ldots, N-1\right\} 的预期奖励 $N$$N$NN 计算为：

where $r_t$$r_t$r_(t)r_{t} is the reward of state $s_t,V\left(s_{(t+N)}\right)$$s_t,V\left(s_{(t+N)}\right)$s_(t),V(s_((t+N)))s_{t}, V\left(s_{(t+N)}\right) is the value of state $s_{(t+N)}$$s_{(t+N)}$s_((t+N))s_{(t+N)}, and $\lambda$$\lambda$lambda\lambda is discount factor. Then, advantage function of actor over critic can be represented as:
其中， $r_t$$r_t$r_(t)r_{t} 表示状态 $s_t,V\left(s_{(t+N)}\right)$$s_t,V\left(s_{(t+N)}\right)$s_(t),V(s_((t+N)))s_{t}, V\left(s_{(t+N)}\right) 的奖励， $s_{(t+N)}$$s_{(t+N)}$s_((t+N))s_{(t+N)} 表示状态 $s_{(t+N)}$$s_{(t+N)}$s_((t+N))s_{(t+N)} 的价值， $\lambda$$\lambda$lambda\lambda 表示折扣因子。 然后，演员相对于批评者的优势函数可以表示为：

From the respect of critic, its target is to minimize $A(a_t,s_t)$$A\left(a_t,s_t\right)$A(a_(t),s_(t))A\left(a_{t}, s_{t}\right) through the gradient descent algorithm as:
从批评的角度来看，其目标是通过梯度下降算法最小化 $A(a_t,s_t)$$A\left(a_t,s_t\right)$A(a_(t),s_(t))A\left(a_{t}, s_{t}\right) ，具体如下：

where $d_{{\theta }_v}$$d_{{\theta }_v}$d_(theta_(v))d_{\theta_{v}} is the gradient of $V$$V$VV. On the other hand, actor’s target is to maximize $A(a(t),s(t))$$A(a(t),s(t))$A(a(t),s(t))A(a(t), s(t)), thus $P$$P$PP 's gradient $d_{{\theta }_p}$$d_{{\theta }_p}$d_(theta_(p))d_{\theta_{p}} is:
其中 $d_{{\theta }_v}$$d_{{\theta }_v}$d_(theta_(v))d_{\theta_{v}} 是 $V$$V$VV 的梯度。另一方面，演员的目标是最大化 $A(a(t),s(t))$$A(a(t),s(t))$A(a(t),s(t))A(a(t), s(t)) ，因此 $P$$P$PP 的梯度 $d_{{\theta }_p}$$d_{{\theta }_p}$d_(theta_(p))d_{\theta_{p}} 为：

where $\log \pi (a_t\mid s_t)$$\log \pi \left(a_t\mid s_t\right)$log pi(a_(t)∣s_(t))\log \pi\left(a_{t} \mid s_{t}\right) is the probability map outputted by $P$$P$PP.
其中 $\log \pi (a_t\mid s_t)$$\log \pi \left(a_t\mid s_t\right)$log pi(a_(t)∣s_(t))\log \pi\left(a_{t} \mid s_{t}\right) 是由 $P$$P$PP 输出的概率图。
In addition, A3C uses asynchronous gradient descent with multiple agents running independently on separate threads, sharing policy and value networks. They gather training data through state transitions, calculate gradients, and update networks asynchronously, allowing for more efficient training and improved policy learning.
此外，A3C 采用异步梯度下降算法，多个代理在独立的线程上运行，共享策略网络和价值网络。它们通过状态转换收集训练数据，计算梯度，并异步更新网络，从而实现更高效的训练和更好的策略学习。

To propose a practical anti-forensics framework, two major challenges need to be addressed. The first one arises from the fact that most forensics methods are black-box systems. As copyright protection and security concerns prevent the disclosure of such methods’ details, only the forensics results are available to users querying these black-box systems. Thus, a practical anti-forensics framework should be developed based on only query results. Building upon this assumption, the simplest query-based attack on black-box inpainting forensics systems can be conducted by adding random noise (perturbation) to inpainting images until the query results indicate that it successfully disrupts the outputs. Fig. 1 illustrates this procedure. However, this type of attack is likely to be query-intensive due to the lack of prior knowledge about the target forensics system and the inefficiency in utilizing query results. Additionally, it may result in visually detectable distortions in inpainting images due to the cumulative effect of excessive random noise.
为了提出一个实用的反取证框架，需要解决两个主要挑战。第一个挑战源于大多数取证方法是黑盒系统。由于版权保护和安全考虑，这些方法的细节无法公开，用户只能通过查询这些黑盒系统获得取证结果。因此，一个实用的反取证框架应基于仅查询结果进行开发。基于此假设，对黑盒图像修复取证系统最简单的查询式攻击可通过向修复图像添加随机噪声（扰动）直至查询结果表明其成功破坏输出结果来实现。图 1 展示了该过程。然而，此类攻击可能因缺乏对目标取证系统的先验知识及查询结果利用效率低下而导致查询密集型。此外，过多的随机噪声的累积效应可能导致修复图像中出现可视化的可检测失真。

Thus, the second challenge entails minimizing the total number of queries $n$$n$nn required while retaining optimal attack performance, which corresponds to build a query-efficient anti-forensics framework. However, the smaller $n$$n$nn usually means a compromised attack performance, thereby defeating the primary objective of the anti-forensics attack. Therefore, it is imperative to balance between visual quality and attack efficacy.
因此，第二个挑战在于在保持最佳攻击性能的同时，尽可能减少所需的查询总数，这对应于构建一个查询高效的反取证框架。然而，较小的查询数量通常意味着攻击性能的下降，从而违背了反取证攻击的主要目标。因此，必须在视觉质量和攻击效果之间取得平衡。

As depicted in Fig. 1, it is evident that at any given time step $(0<t\le n)$$(0<t\le n)$(0 < t <= n)(0<t \leq n), the current inpainting image ${\mathit{X}}_{\mathit{t}}$${\mathit{X}}_{\mathit{t}}$X_(t)\boldsymbol{X}_{\boldsymbol{t}} can be simplified to depend solely on ${\mathit{x}}_{\mathit{t}-\mathbf{1}}$${\mathit{x}}_{\mathit{t}-\mathbf{1}}$x_(t-1)\boldsymbol{x}_{\boldsymbol{t}-\mathbf{1}} and its corresponding perturbation ${\mathit{\xi }}_{t-1}$${\mathit{\xi }}_{t-1}$xi_(t-1)\boldsymbol{\xi}_{t-1}. This dependency can be mathematically expressed based on transition probability $\pi$$\pi$pi\pi as follows:
如图 1 所示，在任意时间步 $(0<t\le n)$$(0<t\le n)$(0 < t <= n)(0<t \leq n) ，当前的插值图像 ${\mathit{X}}_{\mathit{t}}$${\mathit{X}}_{\mathit{t}}$X_(t)\boldsymbol{X}_{\boldsymbol{t}} 可以简化为仅依赖于 ${\mathit{x}}_{\mathit{t}-\mathbf{1}}$${\mathit{x}}_{\mathit{t}-\mathbf{1}}$x_(t-1)\boldsymbol{x}_{\boldsymbol{t}-\mathbf{1}} 及其对应的扰动 ${\mathit{\xi }}_{t-1}$${\mathit{\xi }}_{t-1}$xi_(t-1)\boldsymbol{\xi}_{t-1} 。这种依赖关系可以基于转移概率 $\pi$$\pi$pi\pi 数学表达如下：

This equation confirms that the state transition from ${\mathit{x}}_{\mathit{t}-\mathbf{1}}$${\mathit{x}}_{\mathit{t}-\mathbf{1}}$x_(t-1)\boldsymbol{x}_{\boldsymbol{t}-\mathbf{1}} to ${\mathit{x}}_{\mathit{t}}$${\mathit{x}}_{\mathit{t}}$x_(t)\boldsymbol{x}_{\boldsymbol{t}} satisfies the Markov property, where $t$$t$tt ranges from 1 to $n$$n$nn. Thus, we can model query procedure as a MDP guided by a given policy. And an effective policy is desired to make RLGC query-efficient. We propose a CNN-based policy to accomplish this. Additionally, we employ the A3C algorithm to better optimize our policy network. Prior to this, we need to define the fundamental elements of the RL paradigm used in RLGC.
该方程证实了状态从 ${\mathit{x}}_{\mathit{t}-\mathbf{1}}$${\mathit{x}}_{\mathit{t}-\mathbf{1}}$x_(t-1)\boldsymbol{x}_{\boldsymbol{t}-\mathbf{1}} 到 ${\mathit{x}}_{\mathit{t}}$${\mathit{x}}_{\mathit{t}}$x_(t)\boldsymbol{x}_{\boldsymbol{t}} 的转换满足马尔可夫性质，其中 $t$$t$tt 的取值范围为 1 到 $n$$n$nn 。因此，我们可以将查询过程建模为由给定策略引导的马尔可夫决策问题（MDP）。为了使 RLGC 查询高效，我们需要一个有效的策略。我们提出了一种基于卷积神经网络（CNN）的策略来实现这一目标。此外，我们采用 A3C 算法对策略网络进行进一步优化。在此之前，我们需要定义 RLGC 中使用的强化学习范式的基本要素。

Figure 1: The illustration of querying black-box inpainting .
图 1：黑盒子图像修复的查询示意图。

Environmental model In RLGC, inpainting forensics methods serve as environmental model, with IID-Net(Wu and Zhou 2022) being utilized. IID-Net is selected due to its excellent detection performance and robustness against various image post-processing operations.
环境模型 在 RLGC 中，图像修复取样方法被用作环境模型，其中采用了 IID-Net（Wu 和 Zhou，2022）。IID-Net 被选中是因为其出色的检测性能和对各种图像后处理操作的鲁棒性。

Agent Building upon the multi-threaded asynchronous parallel concept of the A3C framework, we assign an individual agent to each pixel. The objective is to empower each agent to adaptively determine its direction and magnitude of the perturbation by taking into consideration the distribution of neighboring pixels.
基于 A3C 框架的多线程异步并行概念，我们为每个像素分配一个独立的代理。目标是使每个代理能够根据邻近像素的分布，自适应地确定扰动的方向和幅度。

State Our state set $\mathcal{S}$$\mathcal{S}$S\mathcal{S} consists of images set $\mathcal{I}$$\mathcal{I}$I\mathcal{I}, forming a high-dimensional space with the size of ${256}^{(w\times l\times c)}$${256}^{(w\times l\times c)}$256^((w xx l xx c))256^{(w \times l \times c)}. However, it is unnecessary to explore the entire state space as even small perturbations can lead to excellent attack performance. Specifically, given an original inpainting image ${\mathit{X}}_{\mathbf{0}}\in \mathcal{I}$${\mathit{X}}_{\mathbf{0}}\in \mathcal{I}$X_(0)inI\boldsymbol{X}_{\mathbf{0}} \in \mathcal{I}, it serves as the initial state ${\mathit{S}}_{\mathbf{0}}$${\mathit{S}}_{\mathbf{0}}$S_(0)\boldsymbol{S}_{\mathbf{0}}.
状态 我们的状态集 $\mathcal{S}$$\mathcal{S}$S\mathcal{S} 由图像集 $\mathcal{I}$$\mathcal{I}$I\mathcal{I} 组成，形成一个维度为 ${256}^{(w\times l\times c)}$${256}^{(w\times l\times c)}$256^((w xx l xx c))256^{(w \times l \times c)} 的高维空间。然而，无需探索整个状态空间，因为即使是微小的扰动也可能导致出色的攻击性能。具体来说，给定一个原始的图像修复图像 ${\mathit{X}}_{\mathbf{0}}\in \mathcal{I}$${\mathit{X}}_{\mathbf{0}}\in \mathcal{I}$X_(0)inI\boldsymbol{X}_{\mathbf{0}} \in \mathcal{I} ，它作为初始状态 ${\mathit{S}}_{\mathbf{0}}$${\mathit{S}}_{\mathbf{0}}$S_(0)\boldsymbol{S}_{\mathbf{0}} 。

Action RLGC leverages actions as a mean to modulate perturbations for attacking forensics models. To help agents achieve more precise control, we set the magnitude of each perturbation to 1 . For color images with three channels, the image-wise action map $\mathit{A}$$\mathit{A}$A\boldsymbol{A} can be denoted as $\mathit{A}=\{{(a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B)}^{(w\ast h\ast c)}\mid a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B\in$$\mathit{A}=\left\{{\left(a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B\right)}^{(w\ast h\ast c)}\mid a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B\in \right.$A={(a_(i,j,k)^(R),a_(i,j,k)^(G),a_(i,j,k)^(B))^((w**h**c))∣a_(i,j,k)^(R),a_(i,j,k)^(G),a_(i,j,k)^(B)in:}\boldsymbol{A}=\left\{\left(a_{i, j, k}^{R}, a_{i, j, k}^{G}, a_{i, j, k}^{B}\right)^{(w * h * c)} \mid a_{i, j, k}^{R}, a_{i, j, k}^{G}, a_{i, j, k}^{B} \in\right. $\{0,-1,+1\}\}$$\{0,-1,+1\}\}${0,-1,+1}}\{0,-1,+1\}\}, where ( $a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B$$a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B$a_(i,j,k)^(R),a_(i,j,k)^(G),a_(i,j,k)^(B)a_{i, j, k}^{R}, a_{i, j, k}^{G}, a_{i, j, k}^{B} ) are corresponding to $\mathrm{R},\mathrm{G},\mathrm{B}$$\mathrm{R},\mathrm{G},\mathrm{B}$R,G,B\mathrm{R}, \mathrm{G}, \mathrm{B} channel of color images.
动作 RLGC 利用动作作为调节扰动以攻击取证模型的手段。为了帮助代理实现更精确的控制，我们将每个扰动的幅度设置为 1。对于三通道的彩色图像，图像级动作映射 $\mathit{A}$$\mathit{A}$A\boldsymbol{A} 可表示为 $\mathit{A}=\{{(a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B)}^{(w\ast h\ast c)}\mid a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B\in$$\mathit{A}=\left\{{\left(a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B\right)}^{(w\ast h\ast c)}\mid a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B\in \right.$A={(a_(i,j,k)^(R),a_(i,j,k)^(G),a_(i,j,k)^(B))^((w**h**c))∣a_(i,j,k)^(R),a_(i,j,k)^(G),a_(i,j,k)^(B)in:}\boldsymbol{A}=\left\{\left(a_{i, j, k}^{R}, a_{i, j, k}^{G}, a_{i, j, k}^{B}\right)^{(w * h * c)} \mid a_{i, j, k}^{R}, a_{i, j, k}^{G}, a_{i, j, k}^{B} \in\right. $\{0,-1,+1\}\}$$\{0,-1,+1\}\}${0,-1,+1}}\{0,-1,+1\}\} ，其中 ( $a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B$$a_{i,j,k}^R,a_{i,j,k}^G,a_{i,j,k}^B$a_(i,j,k)^(R),a_(i,j,k)^(G),a_(i,j,k)^(B)a_{i, j, k}^{R}, a_{i, j, k}^{G}, a_{i, j, k}^{B} ) 对应于彩色图像的 $\mathrm{R},\mathrm{G},\mathrm{B}$$\mathrm{R},\mathrm{G},\mathrm{B}$R,G,B\mathrm{R}, \mathrm{G}, \mathrm{B} 通道。

State transition The transition of ${\mathit{S}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}}$S_(t)\boldsymbol{S}_{\boldsymbol{t}} to ${\mathit{S}}_{\mathit{t}+\mathbf{1}}$${\mathit{S}}_{\mathit{t}+\mathbf{1}}$S_(t+1)\boldsymbol{S}_{\boldsymbol{t}+\mathbf{1}}, denoted as $T({\mathit{S}}_{\mathit{t}+\mathbf{1}}\mid {\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})$$T\left({\mathit{S}}_{\mathit{t}+\mathbf{1}}\mid {\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}}\right)$T(S_(t+1)∣S_(t),A_(t))T\left(\boldsymbol{S}_{\boldsymbol{t}+\mathbf{1}} \mid \boldsymbol{S}_{\boldsymbol{t}}, \boldsymbol{A}_{\boldsymbol{t}}\right), can be formulated as:
状态转换 ${\mathit{S}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}}$S_(t)\boldsymbol{S}_{\boldsymbol{t}} 到 ${\mathit{S}}_{\mathit{t}+\mathbf{1}}$${\mathit{S}}_{\mathit{t}+\mathbf{1}}$S_(t+1)\boldsymbol{S}_{\boldsymbol{t}+\mathbf{1}} 的转换，记为 $T({\mathit{S}}_{\mathit{t}+\mathbf{1}}\mid {\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})$$T\left({\mathit{S}}_{\mathit{t}+\mathbf{1}}\mid {\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}}\right)$T(S_(t+1)∣S_(t),A_(t))T\left(\boldsymbol{S}_{\boldsymbol{t}+\mathbf{1}} \mid \boldsymbol{S}_{\boldsymbol{t}}, \boldsymbol{A}_{\boldsymbol{t}}\right) ，可表示为：

where ${\mathit{X}}_t$${\mathit{X}}_t$X_(t)\boldsymbol{X}_{t} and ${\mathit{X}}_{\mathit{t}+\mathbf{1}}$${\mathit{X}}_{\mathit{t}+\mathbf{1}}$X_(t+1)\boldsymbol{X}_{\boldsymbol{t}+\boldsymbol{1}} are corresponding to ${\mathit{S}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}}$S_(t)\boldsymbol{S}_{\boldsymbol{t}} and ${\mathit{S}}_{\mathit{t}+\mathbf{1}}.{\mathit{A}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}+\mathbf{1}}.{\mathit{A}}_{\mathit{t}}$S_(t+1).A_(t)\boldsymbol{S}_{\boldsymbol{t}+\boldsymbol{1}} . \boldsymbol{A}_{\boldsymbol{t}} is the action map that agents take at ${\mathit{S}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}}$S_(t)\boldsymbol{S}_{\boldsymbol{t}}.
其中， ${\mathit{X}}_t$${\mathit{X}}_t$X_(t)\boldsymbol{X}_{t} 和 ${\mathit{X}}_{\mathit{t}+\mathbf{1}}$${\mathit{X}}_{\mathit{t}+\mathbf{1}}$X_(t+1)\boldsymbol{X}_{\boldsymbol{t}+\boldsymbol{1}} 分别对应于 ${\mathit{S}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}}$S_(t)\boldsymbol{S}_{\boldsymbol{t}} ，而 ${\mathit{S}}_{\mathit{t}+\mathbf{1}}.{\mathit{A}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}+\mathbf{1}}.{\mathit{A}}_{\mathit{t}}$S_(t+1).A_(t)\boldsymbol{S}_{\boldsymbol{t}+\boldsymbol{1}} . \boldsymbol{A}_{\boldsymbol{t}} 表示代理在 ${\mathit{S}}_{\mathit{t}}$${\mathit{S}}_{\mathit{t}}$S_(t)\boldsymbol{S}_{\boldsymbol{t}} 处执行的动作映射。