Quality-Aware Image-Text Alignment
for Real-World Image Quality Assessment
用于真实世界图像质量评估的质量感知图像文本对齐

CLIP (Contrastive Language-Image Pre-training (CLIP) [26]) is a vision-language model trained on a large-scale dataset with a contrastive loss to semantically align images and corresponding text captions in a shared embedding space. CLIP comprises an image encoder $\psi_{I}$ and a text encoder $\psi_{T}$. Given an image $I$, the image encoder extracts its feature representation $x=\psi_{I}(I)\in\mathbb{R}^{d}$, where $d$ is the CLIP embedding space dimension. For a given text caption $T$, each tokenized word is mapped to the token embedding space $\mathcal{W}$ through a word embedding layer $E_{w}$. Then, the text encoder $\psi_{T}$ is employed to generate the textual feature representation $y=\psi_{T}(E_{w}(T))\in\mathbb{R}^{d}$ from the token embeddings. Thanks to its training strategy, CLIP generates similar representations within the common embedding space for images and text expressing the same concepts.
CLIP（对比语言-图像预训练（CLIP）[26]）是一种视觉语言模型，通过对比损失在大规模数据集上进行训练，以在共享嵌入空间中语义对齐图像和相应的文本说明。CLIP 由一个图像编码器 $\psi_{I}$ 和一个文本编码器 $\psi_{T}$ 组成。给定图像 $I$ ，图像编码器提取其特征表示 $x=\psi_{I}(I)\in\mathbb{R}^{d}$ ，其中 $d$ 是 CLIP 嵌入空间的维度。对于给定的文本说明 $T$ ，每个标记化的单词通过词嵌入层 $E_{w}$ 被映射到标记嵌入空间 $\mathcal{W}$ 。然后，文本编码器 $\psi_{T}$ 用于从标记嵌入中生成文本特征表示 $y=\psi_{T}(E_{w}(T))\in\mathbb{R}^{d}$ 。由于其训练策略，CLIP 在共同嵌入空间中为表达相同概念的图像和文本生成相似的表示。

To make our approach self-supervised, we propose to synthetically degrade unlabeled pristine images using progressively higher levels of intensity. In this way, we can train our model to rank the different versions of each image according to the severity of their degradation. Following [1], we consider 24 distinct degradation types spanning the 7 distortion groups $\mathcal{G}\!=\!\{\mathcal{G}_{1},\ldots,\mathcal{G}_{7}\}$ defined by the KADID [15] dataset. These groups are: 1) Brightness change; 2) Blur; 3) Spatial distortions; 4) Noise; 5) Color distortions; 6) Compression; 7) Sharpness & contrast. Each distortion has $L\!=\!5$ levels of intensity. Figure 2 shows some examples of degraded images for varying degrees of intensity. See the supplementary material for more details on the specific degradation types. Each distortion group is defined as $\mathcal{G}_{i}\!=\!\{\ldots,F^{ij},\ldots\}$, where $i\!\in\!\{1,\ldots,7\}$ indicates the index of the distortion group within $\mathcal{G}$ and $j\!\in\!\{1,\ldots,\left|G_{i}\right|\}$ refers to the index of the degradation type within $\mathcal{G}_{i}$, with $\left|\cdot\right|$ that represents the cardinality.
为了使我们的方法实现自监督，我们建议使用逐渐提高的强度来合成退化未标记的原始图像。通过这种方式，我们可以训练模型根据退化的严重程度对每张图像的不同版本进行排名。根据[1]，我们考虑了 24 种不同的退化类型，涵盖了 KADID[15]数据集中定义的 7 个失真组 $\mathcal{G}\!=\!\{\mathcal{G}_{1},\ldots,\mathcal{G}_{7}\}$ 。这些组包括：1）亮度变化；2）模糊；3）空间畸变；4）噪声；5）颜色失真；6）压缩；7）锐度和对比度。每种失真有 $L\!=\!5$ 强度级别。图 2 展示了某些强度不同的退化图像示例。有关具体退化类型的更多细节，请参阅补充材料。每个失真组定义为 $\mathcal{G}_{i}\!=\!\{\ldots,F^{ij},\ldots\}$ ，其中 $i\!\in\!\{1,\ldots,7\}$ 表示失真组在 $\mathcal{G}$ 中的索引， $j\!\in\!\{1,\ldots,\left|G_{i}\right|\}$ 指退化类型在 $\mathcal{G}_{i}$ 中的索引， $\left|\cdot\right|$ 则代表总数。

Given a training image, we start by extracting a pair of random overlapping crops. Then, we randomly sample $D\!=\!1$ distortion groups and a degradation within each group. We apply the $D$ distortions to both crops using $L\!=\!5$ distinct levels of intensity, resulting in $L$ pairs of equally degraded crops, one for each level. Contrary to [16], we obtain two images for each degree of distortion. When considering two such pairs of crops, we can infer which has a higher quality, based on the corresponding level of degradation. We leverage this information to train our model with a ranking loss.
给定一张训练图像，我们首先提取一对随机重叠的裁剪图。然后，我们随机采样 $D\!=\!1$ 个失真组和每组内的一个退化。我们使用 $D$ 种失真对两个裁剪图应用 $L\!=\!5$ 个不同强度级别，得到 $L$ 对同等退化的裁剪图，每个级别一对。与[16]相反，我们为每个失真程度获得两幅图像。当考虑这两对裁剪图时，我们可以根据相应的退化程度推断哪一个具有更高的质量。我们利用这些信息，通过排名损失来训练我们的模型。

As Fig. 1 shows, CLIP struggles to generate accurate quality-aware image representations that correlate with the severity of the degradation. To address this issue, we propose a quality-aware image-text alignment strategy to fine-tune the CLIP image encoder. The idea of our approach is that given two degraded versions of the same image, a prompt referring to high image quality – such as “Good photo” – should be more similar to the less degraded version. The opposite consideration applies when considering a prompt referring to low image quality, such as “Bad photo”. At the same time, two images with overlapping content and equal degree of degradation should have comparable similarities to such a pair of quality-related prompts, which we refer to as antonym prompts [36]. Note that, given two random images with completely different content, we can not make any assumptions about their relative quality, or, in other words, their similarity to the prompts. Our training strategy leverages multiple pairs of increasingly degraded images to achieve two objectives: O1): we want CLIP to generate consistent representations for images of similar quality, i.e. showing the same amount of distortion; O2): the similarity between each of the antonym prompts and the distinct versions of the images must correlate – in opposite directions – with the corresponding level of degradation.
如图 1 所示，CLIP 在生成与退化程度相关的质量意识图像表示方面存在困难。为了解决这个问题，我们提出了一种质量意识的图像-文本对齐策略来微调 CLIP 图像编码器。我们方法的想法是，给定同一图像的两个退化版本，提到高图像质量的提示——例如“好照片”——应该与退化程度较轻的版本更相似。在考虑与低图像质量相关的提示时，如“坏照片”，则应该考虑相反的情况。同时，具有重叠内容且退化程度相同的两张图片应该与这种一对质量相关的提示具有可比的相似性，我们将这些提示称为反义提示【36】。请注意，考虑到两张内容完全不同的随机图片，我们无法对它们的相对质量做任何假设，或者换句话说，它们与提示的相似性。 我们的训练策略利用多组不断退化的图像对来实现两个目标：O1）：我们希望 CLIP 为质量相似的图像生成一致的表示，即显示相同数量的失真；O2）：每个反义词提示与不同版本图像之间的相似性必须与相应的退化程度在相反方向上相关。

Let $I_{i}^{1}$ and $I_{i}^{2}$ be the $i$-th pair of increasingly degraded crops obtained as detailed in Sec. 3.2, where $i\in\{1,\ldots,L\}$ and $L\!=\!5$ is the number of considered distortion levels. For $i,j\in\{1,\ldots,L\}$ with $j>i$, the $j$-th pair of crops is more degraded than the $i$-th one. Given each pair of crops, we extract the corresponding features through the CLIP image encoder $\psi_{I}$, resulting in $x_{i}^{1}=\psi_{I}(I_{i}^{1})$ and $x_{i}^{2}=\psi_{I}(I_{i}^{2})$. Similarly to [36], we remove the positional embedding to relax the CLIP’s requirement of fixed-size inputs. Let $T_{p}$ and $T_{n}$ be a pair of antonym prompts related to image quality, such as “Good photo” and “Bad photo”. We refer to $T_{p}$ and $T_{n}$ as “positive” and “negative” prompts, respectively. We employ the CLIP text encoder $\psi_{T}$ to extract the text features associated with the prompts, obtaining $t_{p}=\psi_{T}(T_{p})$ and $t_{n}=\psi_{T}(T_{n})$. We normalize both the image and text features to have a unit $L_{2}$-norm.
设 $I_{i}^{1}$ 和 $I_{i}^{2}$ 为第 $i$ 对按逐渐退化的裁剪图像对，在 Sec. 3.2 中详细介绍，其中 $i\in\{1,\ldots,L\}$ 和 $L\!=\!5$ 是所考虑的畸变级别数。对于 $i,j\in\{1,\ldots,L\}$ 和 $j>i$ ，第 $j$ 对裁剪图像比第 $i$ 对更加退化。给定每对裁剪图像，我们通过 CLIP 图像编码器 $\psi_{I}$ 提取相应特征，得到 $x_{i}^{1}=\psi_{I}(I_{i}^{1})$ 和 $x_{i}^{2}=\psi_{I}(I_{i}^{2})$ 。类似于 [36]，我们移除位置嵌入以放松 CLIP 对固定尺寸输入的要求。设 $T_{p}$ 和 $T_{n}$ 为与图像质量相关的反义提示语对，例如“好照片”和“坏照片”。我们分别将 $T_{p}$ 和 $T_{n}$ 称为“正”提示和“负”提示。我们使用 CLIP 文本编码器 $\psi_{T}$ 提取与提示相关的文本特征，得到 $t_{p}=\psi_{T}(T_{p})$ 和 $t_{n}=\psi_{T}(T_{n})$ 。我们将图像和文本特征都归一化以具有单位 $L_{2}$ 范数。

To achieve objective O1, we propose to employ a consistency loss term to guarantee that the similarity between the features of the prompts and those of each of the two images composing each degraded pair is comparable. We assume that two overlapping crops extracted from the same image have a comparable quality [28, 43]. We rely on a margin ranking loss [13, 16, 7] defined as:
为了实现目标 O1，我们建议使用一个一致性损失项来保证提示的特征与构成每个降级对的两个图像的特征之间的相似性是可比的。我们假设从同一图像中提取的两个重叠裁剪具有可比的品质[28, 43]。我们依靠一个边缘排序损失[13, 16, 7]，定义为：

where $c(\cdot)$ stands for the cosine similarity and the margin $m_{cons}$ is a hyperparameter. Intuitively, $m_{cons}$ must be close to 0 to force the similarities between the prompts and each of the two crops to be comparable.
其中 $c(\cdot)$ 表示余弦相似度，边距 $m_{cons}$ 是一个超参数。直观上， $m_{cons}$ 必须接近 0，以迫使提示与两个裁剪之间的相似度具有可比性。

Given the $i$-th level of synthetic degradation, with $i\in\{1,\ldots,L\}$, we assume that the quality of the two distorted crops of the $i$-th pair is higher than that of the two images composing the $(i+1)$-th one [16, 27]. Thus, we enforce that the similarity between the features of the positive prompt and those of two crops is higher than when considering more degraded versions of the two crops. Specifically, we define a margin ranking loss as:
考虑到第 $i$ 级的合成退化程度，给定 $i\in\{1,\ldots,L\}$ ，我们假设第 $i$ 对中的两个失真裁剪图像的质量高于组成第 $(i+1)$ 对的两个图像的质量 [ 16, 27 ]。因此，我们规定正向提示的特征与两个裁剪图像的特征之间的相似性要高于考虑两个裁剪图像更加退化版本的情况。具体地，我们定义一个边距排序损失如下：

where $c(\cdot)$ represents the cosine similarity and the margin $m_{rank}$ is a hyperparameter. The opposite of the consideration made above applies when we take into account the negative prompt. Therefore, we add a loss term to impose that the similarity between the features of the negative prompt and those of two crops is lower than when considering more degraded versions of the two crops:
其中 $c(\cdot)$ 表示余弦相似度， $m_{rank}$ 是一个超参数。当我们考虑负提示时，上述考虑的相反情况适用。因此，我们添加一个损失项，以强制负提示的特征与两个裁剪的特征之间的相似度低于考虑两个裁剪的更多降级版本时的情况：

with $c(\cdot)$ and $m_{rank}$ defined as above. Intuitively, we need to set $m_{rank}\!>>\!0$ as we aim for a noticeable difference between the similarities of the prompts and the increasingly degraded versions of the two crops. The combination of $\mathcal{L}_{pos}$ and $\mathcal{L}_{neg}$ allows us to achieve objective O2.
如上所述定义了 $c(\cdot)$ 和 $m_{rank}$ 。直观而言，我们需要设定 $m_{rank}\!>>\!0$ ，因为我们希望在提示的相似性与两个剪裁的逐渐退化版本之间产生显著差异。 $\mathcal{L}_{pos}$ 和 $\mathcal{L}_{neg}$ 的结合使我们能够实现目标 O2。

The final training loss used to fine-tune the CLIP image encoder is given by:
最后用于微调 CLIP 图像编码器的训练损失为：

where $\lambda_{cons}$, $\lambda_{pos}$, and $\lambda_{neg}$ represent the loss weights. Figure 3 shows an overview of our training strategy. Given that we do not employ any labeled MOS, our approach is both self-supervised and opinion-unaware. Thanks to the proposed training strategy, CLIP learns an image-text alignment not based on high-level semantics, but rather on low-level image characteristics, such as noise and blur. As a result, QualiCLIP generates representations that correlate with the intrinsic quality of the images, as shown in Fig. 1.
其中 $\lambda_{cons}$ 、 $\lambda_{pos}$ 和 $\lambda_{neg}$ 代表损失权重。图 3 展示了我们训练策略的概述。由于我们没有使用任何标记的 MOS，我们的方法既是自监督的也是意见无关的。得益于提出的训练策略，CLIP 学习到了不基于高级语义，而是基于低级图像特征（如噪声和模糊）的图像文本对齐。因此，QualiCLIP 生成的表示与图像的内在质量相关，如图 1 所示。

At inference time, given an image $I$, we extract its features $x$ using the CLIP image encoder. Then, we compute the cosine similarity between $x$ and the features $t_{p}$ and $t_{n}$ of the antonym prompts, resulting in $s_{p}$ and $s_{n}$. Finally, similar to [36], we obtain the final quality score $q\!\in\![0,1]$ by using the softmax:
在推理时，给定图像 $I$ ，我们使用 CLIP 图像编码器提取其特征 $x$ 。然后，我们计算 $x$ 与反义提示的特征 $t_{p}$ 和 $t_{n}$ 之间的余弦相似度，得到 $s_{p}$ 和 $s_{n}$ 。最后，类似于[36]，我们使用 softmax 获得最终的质量分数 $q\!\in\![0,1]$ 。

Figure 4 provides an overview of the final quality score computation. Note that, since we keep the CLIP text encoder weights frozen, we need to compute the text features of the antonym prompts just once, and we can use them both for training and inference. Therefore, at inference time the computational cost of our method is the same as an image-encoder-only model with the same backbone.
图 4 概述了最终质量评分的计算。请注意，由于我们保持 CLIP 文本编码器的权重不变，因此我们只需计算一次反义词提示的文本特征，并能在训练和推理中使用它们。因此，在推理阶段，我们的方法与使用相同骨干的仅图像编码器模型具有相同的计算成本。

We recall that the authors of GRepQ [32] use a strategy similar to the one shown in Fig. 4 to predict the quality of all the images comprising a training batch. Then, they divide the images into a high-quality and a low-quality group. Finally, they fine-tune the CLIP image encoder with a contrastive loss by maximizing the intra-group similarity and minimizing the inter-group one. In contrast, we consider multiple pairs of increasingly synthetically degraded crops. We rely on $\mathcal{L}_{cons}$ to fine-tune the CLIP image encoder so that it generates consistent representations for images exhibiting the same degree of distortion. At the same time, we use $\mathcal{L}_{pos}$ and $\mathcal{L}_{neg}$ to train our model to rank the similarity between two antonym prompts and the images, according to their level of degradation. Thanks to our training strategy, CLIP yields more accurate quality-aware image representations.
我们回顾一下，GRepQ [32] 的作者使用了一种类似于图 4 所示的策略来预测训练批次中所有图像的质量。然后，他们将图像分为高质量和低质量组。最后，他们通过最大化组内相似性和最小化组间相似性来用对比损失微调 CLIP 图像编码器。相比之下，我们考虑多个对比度逐渐合成退化的图块。我们依赖 $\mathcal{L}_{cons}$ 来微调 CLIP 图像编码器，以便为展现相同程度失真的图像生成一致的表现。同时，我们使用 $\mathcal{L}_{pos}$ 和 $\mathcal{L}_{neg}$ 训练我们的模型，根据图像的退化水平排列两个反义提示之间以及图像之间的相似性。得益于我们的训练策略，CLIP 能够产生更准确的质量感知图像表现。

We train our model using the 140K pristine images of the KADIS dataset [15]. Given a training image, we synthetically degrade it by employing the strategy detailed in Sec. 3.2. We validate and test the proposed approach on IQA datasets with synthetic and authentic distortions, respectively. These datasets contain a set of degraded images annotated with human judgments of picture quality in the form of a Mean Opinion Score (MOS). For validation, we consider two synthetically degraded datasets: LIVE [30] and TID2013 [25]. LIVE stems from 29 reference images, each distorted with 5 types of degradation at 5 levels of intensity, resulting in 779 images. Conversely, TID2013 contains 3000 images degraded with 24 distinct distortions at 5 levels of intensity, with 25 reference images as the base. For testing, we consider four datasets with authentic distortions: KonIQ [10], CLIVE [6], FLIVE [39], and SPAQ [5]. KonIQ contains 10K images sampled from the YFCC100M [34] database. CLIVE consists of 1162 images captured with a wide range of mobile devices. FLIVE is the largest existing dataset for NR-IQA and is composed of about 40K real-world images. SPAQ comprises 11K high-resolution photos taken with several smartphones. Following [5], we resize the SPAQ images so that the shorter side is 512.
我们使用 KADIS 数据集[15]中的 140K 原始图像训练我们的模型。给定一个训练图像，我们通过使用第 3.2 节中详细介绍的策略对其进行合成降级。我们在分别具有合成和真实失真的 IQA 数据集上验证和测试所提议的方法。这些数据集包含一组降级图像，并附有人类对图像质量的判断，以平均意见得分（MOS）的形式进行标注。对于验证，我们考虑两个合成降级数据集：LIVE[30]和 TID2013[25]。LIVE 来自 29 个参考图像，每个图像使用 5 种类型的降级在 5 个强度级别上进行处理，共得到 779 张图像。相反，TID2013 包含 3000 张图像，经过 24 种不同失真在 5 个强度级别下降级，以 25 张参考图像为基础。对于测试，我们考虑四个真实失真数据集：KonIQ[10]、CLIVE[6]、FLIVE[39]和 SPAQ[5]。KonIQ 包含从 YFCC100M[34]数据库中采样的 10K 图像。CLIVE 由 1162 张使用各种移动设备拍摄的图像组成。FLIVE 是现有的最大 NR-IQA 数据集，由约 40K 张真实世界图像组成。 SPAQ 包含 11K 张用多款智能手机拍摄的高分辨率照片。根据[ 5]，我们将 SPAQ 图像的短边调整为 512。

We rely on a ResNet50 [9] as the backbone for CLIP. Similar to [36], we remove the positional embedding from the encoder to allow our model to take images of any resolution as input. The dimension $d$ of the CLIP embedding space is 1024. Differently from [32], we do not train a projector head on top of the CLIP image encoder. We keep the CLIP text encoder frozen. Similar to [38, 37], we employ multiple pairs of antonym prompts. We train our model for 3 epochs using an AdamW [17] optimizer with a weight decay and a learning rate of $1e\!-\!2$ and $1e\!-\!9$, respectively. During training, we employ a patch size of 224 and a batch size of 16. We set the margins $m_{cons}$ in Eq. 1 and $m_{rank}$ in Eqs. 2 and 3 to $2.5e\!-\!3$ and $6.75e\!-\!2$, respectively. The loss weights $\lambda_{cons}$, $\lambda_{pos}$ and $\lambda_{neg}$ in Eq. 4 are all equal to 1. At inference time, our model takes the whole image as input and outputs a single quality score.
我们依赖于 ResNet50 [9]作为 CLIP 的主干网络。与[36]类似，我们从编码器中移除位置嵌入，以允许我们的模型接受任何分辨率的图像作为输入。CLIP 嵌入空间的维度 $d$ 为 1024。不同于[32]，我们没有在 CLIP 图像编码器之上训练投射头。我们保持 CLIP 文本编码器冻结。类似于[38, 37]，我们使用多对反义词提示。我们使用 AdamW [17]优化器进行 3 轮训练，权重衰减和学习率分别为 $1e\!-\!2$ 和 $1e\!-\!9$ 。在训练期间，我们使用 224 的补丁尺寸和 16 的批量大小。我们将方程 1 中的边距 $m_{cons}$ 和方程 2 及 3 中的边距设置为 $2.5e\!-\!3$ 和 $6.75e\!-\!2$ 。方程 4 中的损失权重 $\lambda_{cons}$ 、 $\lambda_{pos}$ 和 $\lambda_{neg}$ 均等于 1。在推理时，我们的模型接受整个图像作为输入，并输出单个质量分数。

We compare our approach to state-of-the-art methods in two different settings: zero-shot and cross-dataset. Note that our method remains consistent across both settings; the only variation lies in the baselines we compare against. For a fair comparison, we compute the results of each baseline using our evaluation protocol by employing the official pre-trained model if available or training it from scratch using the original hyperparameters. In the zero-shot setting, we only consider opinion-unaware methods [24, 40, 4, 36] and approaches that, with slight modifications, can function without requiring MOS [23, 28, 1, 32]. In particular, we follow the original paper for GRepQ [32], while for methods based on a linear regressor [23, 28, 1], such as CONTRIQUE [23], we use a NIQE-style framework on the extracted image features, similar to [4]. We evaluate the performance using the full datasets for testing. In the cross-dataset setting, we compare with supervised methods [33, 7, 23, 28, 1, 36, 32] using testing datasets different from the training one, simulating real-world scenarios. Here, we use the full datasets both for training and testing.
我们在两种不同的环境中将我们的方法与现有最先进的方法进行比较：零样本和跨数据集。请注意，我们的方法在这两种环境中保持一致；唯一的变化在于我们所比较的基线。为了公平比较，我们在评估过程中使用官方预训练模型（如果可用）或使用原始超参数从头开始训练来计算每个基线的结果。在零样本环境中，我们仅考虑意见无关的方法 [24, 40, 4, 36] 以及经过轻微修改后能够在不需要 MOS 的情况下运行的方法 [23, 28, 1, 32]。特别地，对于 GRepQ [32] 方法，我们遵循原始论文，而对于基于线性回归器的方法 [23, 28, 1]，如 CONTRIQUE [23]，我们在提取的图像特征上使用类似于 [4] 的 NIQE 风格框架。我们使用完整的数据集进行测试来评估性能。在跨数据集环境中，我们与监督方法 [33, 7, 23, 28, 1, 36, 32] 进行比较，使用与训练数据集不同的测试数据集，模拟现实世界情景。在这里，我们同时使用完整的数据集进行训练和测试。

We conduct ablations studies on the LIVE and TID2013 synthetic datasets to evaluate the individual contribution of each component of our approach.
我们在 LIVE 和 TID2013 合成数据集上进行了消融研究，以评估我们方法中每个组件的独立贡献。

We evaluate the robustness and the explainability of our model through the gMAD [21] competition and a gradCAM [29] visualization, respectively. We report additional experiments in the supplementary material.
我们通过 gMAD [21]竞赛和 gradCAM [29]可视化分别评估了我们模型的鲁棒性和可解释性。我们在补充材料中报告了额外的实验。

We report additional quantitative results, following the evaluation protocol detailed in Sec. 4.3. In particular, we consider synthetic datasets in the zero-shot setting, while we employ the CLIVE dataset [6] to train the baselines in the cross-dataset setting.

We compare the robustness of QualiCLIP with CLIP-IQA [36] by conducting the group maximum differentiation (gMAD) competition [21]. We provide more details on gMAD in Sec. 4.5. Figure S7 shows the results. When QualiCLIP is fixed (Fig. 7(a)), CLIP-IQA struggles to identify image pairs with an evident quality gap. In contrast, when QualiCLIP operates as the attacker (Fig. 7(b)), it successfully highlights the failures of CLIP-IQA by finding image pairs displaying significantly different quality. This result shows that our approach demonstrates greater robustness than CLIP-IQA.

We compare the image representations generated by QualiCLIP and CLIP-IQA [36] via a t-SNE [22] visualization. Following [32], we consider images from the CLIVE [6] dataset with very high or very low quality. In particular, we take into account images with a labeled MOS greater than 75 and lower than 25, respectively. Figure S8 shows the results. We observe that the representations of high- and low-quality images obtained by the proposed approach (Fig. S8) correspond to more easily separable clusters compared to those of CLIP-IQA (Fig. S8), which are more intertwined. This result confirms that QualiCLIP generates image representations that better correlate with their intrinsic quality.

Following [37, 38], we employ multiple pairs of antonym prompts during training and inference. In particular, we use: 1) “Good/Bad photo”; 2) “Good/Bad picture”; 3) “High-resolution/Low-resolution image”; 4) “High-quality/Low-quality image”; 5) “Sharp/Blurry image”; 6) “Sharp/Blurry edges”; 7) “Noise-free/Noisy image”. We average the similarities between the images and the pairs of prompts. As we keep the CLIP text encoder fixed, computing the text features of the prompts is a one-time requirement. Subsequently, we can employ them both for training and inference.

As detailed in Sec. 3.2, during training we synthetically degrade pristine images with increasing intensity levels to make our approach self-supervised. Specifically, similar to [1] we consider 24 distinct distortion types divided into the 7 degradation groups defined by the KADID [15] dataset. Each degradation has 5 degrees of progressively higher intensity. We report an example for all the intensity levels of each distortion belonging to the degradation groups in Figs. S9, S10, S11, S12, S13, S14 and S15. Each distortion is described as follows:

1. 1.

   Brightness change:

   • •

     Brighten: applies a sequence of color space transformations, curve adjustments, and blending operations to increase the brightness of the image;

   • •

     Darken: similar to the brighten operation, but reduces the brightness instead of increasing it;

   • •

     Mean shift: adjusts the average intensity of image pixels by adding a constant value to all pixel values. Then, it constrains the resulting values to stay within the original image range;

2. 2.

   Blur:

   • •

     Gaussian blur: applies a Gaussian kernel filter to each image pixel;

   • •

     Lens blur: applies a circular kernel filter to each image pixel;

   • •

     Motion blur: applies a linear motion blur kernel to each image pixel, simulating the effect of either a moving camera or a moving object in the scene. This results in the image appearing blurred in the direction of the motion;

3. 3.

   Spatial distortions:

   • •

     Jitter: randomly displaces image data by applying small offsets to warp each pixel;

   • •

     Non-eccentricity patch: randomly selects patches from the image and places them in random neighboring positions;

   • •

     Pixelate: employs a combination of downscaling and upscaling operations using nearest-neighbor interpolation;

   • •

     Quantization: quantizes the image into $N$ uniform levels. The quantization thresholds are dynamically computed using Multi-Otsu’s method [14];

   • •

     Color block: randomly superimposes uniformly colored square patches onto the image;

4. 4.

   Noise:

   • •

     White noise: adds Gaussian white noise to the image;

   • •

     White noise in color component: transforms the image to the YCbCr color space and then adds Gaussian white noise to each channel;

   • •

     Impulse noise: adds salt and pepper noise to the image;

   • •

     Multiplicative noise: adds speckle noise to the image;

5. 5.

   Color distortions:

   • •

     Color diffusion: transforms the image to the LAB color space and then applies Gaussian blur to each channel;

   • •

     Color shift: randomly shifts the green channel and then blends it into the original image, masking it with the normalized gradient magnitude of the original image;

   • •

     Color saturation 1: transforms the image to the HSV color space and then scales the saturation channel by a factor;

   • •

     Color saturation 2: transforms the image to the LAB color space and then scales each color channel by a factor;

6. 6.

   Compression:

   • •

     JPEG2000: applies the standard JPEG2000 compression to the image;

   • •

     JPEG: applies the standard JPEG compression to the image;

7. 7.

   Sharpness & contrast:

   • •

     High sharpen: applies unsharp masking to sharpen the image in the LAB color space;

   • •

     Nonlinear contrast change: applies a nonlinear tone mapping operation to adjust the contrast of the image;

   • •

     Linear contrast change: applies a linear tone mapping operation to adjust the contrast of the image;