DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs
DeepFuse：一种针对极端曝光图像对的深度无监督曝光融合方法

The learning ability of CNN is heavily influenced by right choice of architecture and loss function. A simple and naive architecture is to have a series of convolutional layers connected in sequential manner. The input to this architecture would be exposure image pairs stacked in third dimension. Since the fusion happens in the pixel domain itself, this type of architecture does not make use of feature learning ability of CNNs to a great extent.
卷积神经网络的学习能力很大程度上取决于架构设计与损失函数的合理选择。一种简单直接的架构设计是采用多个卷积层顺序连接的结构。该架构的输入为沿第三维度堆叠的曝光图像对。由于融合过程直接作用于像素域，此类架构未能充分发挥卷积神经网络的特征学习能力。

The proposed network architecture for image fusion is illustrated in Fig. 2. The proposed architecture has three components: feature extraction layers, a fusion layer and reconstruction layers. As shown in Fig. 2, the under-exposed and the over-exposed images ($Y_{1}$ and $Y_{2}$) are input to separate channels (channel 1 consists of C11 and C21 and channel 2 consists of C12 and C22). The first layer (C11 and C12) contains 5 $\times$ 5 filters to extract low-level features such as edges and corners. The weights of pre-fusion channels are tied, C11 and C12 (C21 and C22) share same weights. The advantage of this architecture is three fold: first, we force the network to learn the same features for the input pair. That is, the F11 and F21 are same feature type. Hence, we can simply combine the respective feature maps via fusion layer. Meaning, the first feature map of image 1 (F11) and the first feature map of image 2 (F21) are added and this process is applied for remaining feature maps as well. Also, adding the features resulted in better performance than other choices of combining features (see Table 1). In feature addition, similar feature types from both images are fused together. Optionally one can choose to concatenate features, by doing so, the network has to figure out the weights to merge them. In our experiments, we observed that feature concatenation can also achieve similar results by increasing the number of training iterations, increasing number of filters and layers after C3. This is understandable as the network needs more number of iterations to figure out appropriate fusion weights. In this tied-weights setting, we are enforcing the network to learn filters that are invariant to brightness changes. This is observed by visualizing the learned filters (see Fig. 8). In case of tied weights, few high activation filters have center surround receptive fields (typically observed in retina). These filters have learned to remove the mean from neighbourhood, thus effectively making the features brightness invariant. Second, the number of learnable filters is reduced by half. Third, as the network has low number of parameters, it converges quickly. The obtained features from C21 and C22 are fused by merge layer. The result of fuse layer is then passed through another set of convolutional layers (C3, C4 and C5) to reconstruct final result ($Y_{fused}$) from fused features.
图 2 展示了本文提出的图像融合网络架构。该架构包含三个核心组件：特征提取层、融合层和重建层。如图 2 所示，欠曝光图像（ $Y_{1}$ ）与过曝光图像（ $Y_{2}$ ）分别输入两个独立通道（通道 1 包含 C11 与 C21，通道 2 包含 C12 与 C22）。第一层（C11 和 C12）采用 5×5 滤波器提取边缘与角点等底层特征。预融合通道采用权重共享机制，即 C11 与 C12（以及 C21 与 C22）具有相同权重。该架构具有三重优势：首先，强制网络对输入图像对学习相同特征类型，确保 F11 与 F21 属于同类特征，从而可直接通过融合层合并对应特征图——即将图像 1 的首个特征图（F11）与图像 2 的首个特征图（F21）相加，该操作同样适用于其余特征图。实验表明，特征相加的融合方式相较其他组合策略能获得更优性能（参见表 1）。在特征相加过程中，两幅图像的同类特征得以有效融合。 可选地，可以选择通过特征拼接的方式，此时网络需要自行学习特征融合的权重。在实验中我们发现，通过增加训练迭代次数、在 C3 层后增加滤波器数量与网络层数，特征拼接同样能取得相近效果。这可以理解，因为网络需要更多迭代次数来学习合适的融合权重。在这种权重共享的设置下，我们强制网络学习对亮度变化具有不变性的滤波器。通过可视化已学习的滤波器（见图 8）可以观察到：在权重共享情况下，少数高激活滤波器具有中心-外周感受野（视网膜中常见特性），这些滤波器学会了消除邻域均值，从而有效实现特征亮度不变性。其次，可训练滤波器数量减少了一半。再者，由于网络参数量较少，收敛速度更快。最终，C21 与 C22 层提取的特征通过融合层进行合并。 融合层的结果随后通过另一组卷积层（C3、C4 和 C5）进行处理，从融合特征中重建最终结果（ $Y_{fused}$ ）。

We have collected 25 exposure stacks that are available publicly [1]. In addition to that, we have curated 50 exposure stacks with different scene characteristics. The images were taken with standard camera setup and tripod. Each scene consists of 2 low dynamic range images with $\pm 2$ EV difference. The input sequences are resized to 1200 $\times$ 800 dimensions. We give priority to cover both indoor and outdoor scenes. From these input sequences, 30000 patches of size 64 $\times$64 were cropped for training. We set the learning rate to $10^{-4}$ and train the network for 100 epochs with all the training patches being processed in each epoch.
我们收集了 25 组公开可用的曝光堆栈[1]，此外还整理了 50 组具有不同场景特征的曝光堆栈。这些图像均采用标准相机配置和三脚架拍摄，每个场景包含 2 张动态范围较低且曝光值相差 $\pm 2$ 档的图像。输入序列统一调整为 1200 $\times$ 800 像素尺寸，并优先覆盖室内外多种场景。从这些输入序列中，我们裁剪出 30000 个 64 $\times$ 64 尺寸的图块用于训练，设置学习率为 $10^{-4}$ ，并以所有训练图块每轮完整遍历的方式训练网络 100 个周期。

We follow the standard cross-validation procedure to train our model and test the final model on a disjoint test set to avoid over-fitting. While testing, the trained CNN takes the test image sequence and generates the luminance channel ($Y_{fused}$) of fused image. The chrominance components of fused image, $Cb_{fused}$ and $Cr_{fused}$, are obtained by weighted sum of input chrominance channel values.
我们采用标准交叉验证流程训练模型，并在独立测试集上评估最终模型以防止过拟合。测试时，训练好的 CNN 网络接收测试图像序列并生成融合图像的亮度通道( $Y_{fused}$ )，而融合图像的色度分量 $Cb_{fused}$ 和 $Cr_{fused}$ 则通过输入色度通道值的加权求和获得。

The crucial structural details of the image tend to be present mainly in $Y$ channel. Thus, different fusion strategies are followed in literature for $Y$ and $Cb$/$Cr$ fusion ([18], [24], [26]). Moreover, MEF SSIM loss is formulated to compute the score between 2 gray-scale ($Y$) images. Thus, measuring MEF SSIM for $Cb$ and $Cr$ channels may not be meaningful. Alternately, one can choose to fuse RGB channels separately using different networks. However, there is typically a large correlation between RGB channels. Fusing RGB independently fails to capture this correlation and introduces noticeable color difference. Also, MEF-SSIM is not designed for RGB channels. Another alternative is to regress RGB values in a single network, then convert them to a $Y$ image and compute MEF SSIM loss. Here, the network can focus more on improving $Y$ channel, giving less importance to color. However, we observed spurious colors in output which were not originally present in input.
图像的关键结构细节往往主要存在于 $Y$ 通道中。因此，文献中对 $Y$ 与 $Cb$ / $Cr$ 融合采用了不同策略（[18]、[24]、[26]）。此外，MEF SSIM 损失函数专为计算两幅灰度（ $Y$ ）图像间的评分而设计，故对 $Cb$ 和 $Cr$ 通道进行 MEF SSIM 评估可能缺乏意义。替代方案之一是采用不同网络分别融合 RGB 通道，但 RGB 通道间通常存在强相关性，独立融合会丢失这种关联并导致明显色差。况且 MEF-SSIM 本非为 RGB 通道设计。另一替代方案是通过单一网络回归 RGB 值，再转换为 $Y$ 图像计算 MEF SSIM 损失——此时网络可侧重优化 $Y$ 通道而弱化色彩处理。但我们发现该方法会在输出中产生输入图像原本不存在的伪色。

We follow the procedure used by Prabhakar et al. [18] for chrominance channel fusion. If $x_{1}$ and $x_{2}$ denote the $Cb$ (or $Cr$) channel value at any pixel location for image pairs, then the fused chrominance value $x$ is obtained as follows,
我们采用 Prabhakar 等人[18]提出的色度通道融合方法。若 $x_{1}$ 和 $x_{2}$ 表示图像对中任意像素位置的 $Cb$ （或 $Cr$ ）通道值，则融合后的色度值 $x$ 按以下方式计算：

The fused chrominance value is obtained by weighing two chrominance values with $\tau$ subtracted value from itself. The value of $\tau$ is chosen as 128. The intuition behind this approach is to give more weight for good color components and less for saturated color values. The final result is obtained by converting {$Y_{fused}$, $Cb_{fused}$, $Cr_{fused}$} channels into RGB image.
融合色度值通过对两个色度值进行加权获得，其中权重为 $\tau$ 与其自身相减的值。 $\tau$ 的取值设定为 128。该方法的核心思想是对优质色彩成分赋予更大权重，而对饱和色彩值降低权重。最终通过将{ $Y_{fused}$ , $Cb_{fused}$ , $Cr_{fused}$ }通道转换至 RGB 空间得到合成图像。

So far, we have discussed on training CNN model in unsupervised manner. One interesting variant of that would be to train the CNN model with results of other state-of-art methods as ground truth. This experiment can test the capability of CNN to learn complex fusion rules from data itself without the help of MEF SSIM loss function. The ground truth is selected as best of Mertens [16] and GFF [10] methods based on MEF SSIM score²²2In a user survey conducted by Ma et al. [15], Mertens and GFF results are ranked better than other MEF algorithms. The choice of loss function to calculate error between ground truth and estimated output is very crucial for training a CNN in supervised fashion. The Mean Square Error or $\ell_{2}$ loss function is generally chosen as default cost function for training CNN. The $\ell_{2}$ cost function is desired for its smooth optimization properties. While $\ell_{2}$ loss function is better suited for classification tasks, they may not be a correct choice for image processing tasks [29]. It is also a well known phenomena that MSE does not correlate well with human perception of image quality [27]. In order to obtain visually pleasing result, the loss function should be well correlated with HVS, like Structural Similarity Index (SSIM) [27]. We have experimented with different loss functions such as $\ell_{1}$, $\ell_{2}$ and SSIM.
迄今为止，我们已探讨了以无监督方式训练 CNN 模型的方法。其中一个有趣的变体是采用其他先进方法的结果作为基准真值来训练 CNN 模型。该实验可验证 CNN 在无需 MEF SSIM 损失函数辅助的情况下，从数据中学习复杂融合规则的能力。基于 MEF SSIM 评分 ²²2In a user survey conducted by Ma et al. [15], Mertens and GFF results are ranked better than other MEF algorithms ，我们选取 Mertens[16]与 GFF[10]方法中的最优结果作为基准真值。在监督式 CNN 训练中，用于计算基准真值与估计输出间误差的损失函数选择至关重要。均方误差（MSE）或 $\ell_{2}$ 损失函数通常作为默认代价函数，其平滑优化特性使 $\ell_{2}$ 代价函数备受青睐。虽然 $\ell_{2}$ 损失函数更适用于分类任务，但对于图像处理任务可能并非最佳选择[29]。众所周知，MSE 与人类视觉感知的图像质量相关性较弱[27]。为获得视觉感知良好的结果，损失函数应与人类视觉系统（HVS）高度相关，例如结构相似性指数（SSIM）[27]。 我们尝试了不同的损失函数，如 $\ell_{1}$ 、 $\ell_{2}$ 和结构相似性指数（SSIM）。

The fused image appear blurred when the CNN was trained with $\ell_{2}$ loss function. This effect termed as regression to mean, is due to the fact that $\ell_{2}$ loss function compares the result and ground truth in a pixel by pixel manner. The result by $\ell_{1}$ loss gives sharper result than $\ell_{2}$ loss but it has halo effect along the edges. Unlike $\ell_{1}$ and $\ell_{2}$, results by CNN trained with SSIM loss function are both sharp and artifact-free. Therefore, SSIM is used as loss function to calculate error between generated output and ground truth in this experiment.
当 CNN 使用 $\ell_{2}$ 损失函数训练时，融合图像会出现模糊现象。这种被称为"均值回归"的效应源于 $\ell_{2}$ 损失函数以逐像素方式比较结果与真实值。使用 $\ell_{1}$ 损失函数得到的结果比 $\ell_{2}$ 损失函数更清晰，但边缘存在光晕效应。与 $\ell_{1}$ 和 $\ell_{2}$ 不同，采用 SSIM 损失函数训练的 CNN 结果既清晰又无伪影。因此本实验采用 SSIM 作为损失函数来计算生成输出与真实值之间的误差。

The quantitative comparison between DeepFuse baseline and unsupervised method is shown in Table 2. The MEF SSIM scores in Table 2 shows the superior performance of DeepFuse unsupervised over baseline method in almost all test sequences. The reason is due to the fact that for baseline method, the amount of learning is upper bound by the other algorithms, as the ground truth for baseline method is from Merterns et al. [16] or Li et al. [10]. We see from Table 2 that the baseline method does not exceed both of them.
表 2 展示了 DeepFuse 基线方法与无监督方法的定量对比结果。表中 MEF SSIM 评分表明，在几乎所有测试序列中，DeepFuse 无监督方法均优于基线方法。这是由于基线方法的学习效果受限于其他算法，因其真值数据来源于 Merterns 等人[16]或 Li 等人[10]的研究。从表 2 可见，基线方法的性能始终未能超越这两个参照方法。

The idea behind this experiment is to combine advantages of all previous methods, at the same time avoid shortcomings of each. From Fig. 3, we can observe that though DF-baseline is trained with results of other methods, it can produce results that do not have any artifacts observed in other results.
本实验的核心思想在于综合先前所有方法的优势，同时规避各自的缺陷。从图 3 可以观察到，尽管 DF-baseline 模型是使用其他方法的结果进行训练的，但其生成的结果并未出现其他方法中常见的伪影现象。

Comparison with Mertens et al.: Mertens et al. [16] is a simple and effective weighting based image fusion technique with multi resolution blending to produce smooth results. However, it suffers from following shortcomings: (a) it picks “best” parts of each image for fusion using hand crafted features like saturation and well-exposedness. This approach would work better for image stacks with many exposure images. But for exposure image pairs, it fails to maintain uniform brightness across whole image. Compared to Mertens et al., DeepFuse produces images with consistent and uniform brightness across whole image. (b) Mertens et al. does not preserve complete image details from under exposed image. In Fig. 4(d), the details of the tile area is missing in Mertens et al.’s result. The same is the case in Fig. 4(j), the fine details of the lamp are not present in the Mertens et al. result. Whereas, DeepFuse has learned filters that extract features like edges and textures in C1 and C2, and preserves finer structural details of the scene.
与 Mertens 等人的方法比较：Mertens 等人[16]提出了一种基于权重分配的多分辨率融合技术，该方法简单有效且能生成平滑结果。但其存在以下不足：(a)该方法通过手工设计特征（如饱和度和曝光良好度）从每幅图像中选取"最佳"部分进行融合。这种策略在曝光图像数量较多时效果较好，但对于曝光图像对而言，难以维持整幅图像的亮度均匀性。相比之下，DeepFuse 生成的图像在整个画面中保持亮度一致且均匀。(b)Mertens 等人的方法无法完整保留欠曝光图像中的细节。如图 4(d)所示，瓦片区域的细节在其结果中缺失；同样在图 4(j)中，灯具的精细结构也未能在其结果中呈现。而 DeepFuse 通过学习得到的滤波器，能够在 C1 和 C2 层提取边缘和纹理等特征，从而保留场景更精细的结构细节。

Comparison with Li et al. [9] [10]: It can be noted that, similar to Mertens et al. [16], Li et al. [9] [10] also suffers from non-uniform brightness artifact (Fig. 5). In contrast, our algorithm provides a more pleasing image with clear texture details.
与 Li 等人[9][10]的对比：可以注意到，与 Mertens 等人[16]类似，Li 等人[9][10]的方法同样存在亮度不均匀的伪影问题（图 5）。相比之下，我们的算法生成的图像视觉效果更佳，纹理细节清晰。

Comparison with Shen et al. [23]: The results generated by Shen et al. show contrast loss and non-uniform brightness distortions (Fig. 5). In Fig. 5(e1), the brightness distortion is present in the cloud region. The cloud regions in between balloons appear darker compared to other regions. This distortion can be observed in other test images as well in Fig. 5(e2). However, the DeepFuse (Fig. 5(f1) and (f2) ) have learnt to produce results without any of these artifacts.
与 Shen 等人[23]的对比：Shen 等人方法生成的结果存在对比度损失和亮度不均匀失真（图 5）。在图 5(e1)中，云层区域出现了亮度失真，气球之间的云区比其他区域显得更暗。这种失真现象在图 5(e2)的其他测试图像中同样可见。而 DeepFuse 方法（图 5(f1)和(f2)）则能够生成完全没有这些伪影的结果。

Comparison with Ma et al. [14]: Fig. 6 and 7 shows comparison between results of Ma et al. and DeepFuse for Lighthouse and Table sequences. Ma et al. proposed a patch based fusion algorithm that fuses patches from input images based on their patch strength. The patch strength is calculated using a power weighting function on each patch. This method of weighting would introduce unpleasant halo effect along edges (see Fig. 6 and 7).
与 Ma 等人[14]的对比：图 6 和图 7 展示了 Lighthouse 和 Table 序列在 Ma 等人方法与 DeepFuse 方法下的结果比较。Ma 等人提出了一种基于图像块的融合算法，该算法根据图像块的强度对输入图像的块进行融合。块强度通过每个块的幂加权函数计算得出。这种加权方法会在边缘处产生不自然的光晕效应（见图 6 和图 7）。

Comparison with Raman et al. [20]: Fig. 3(f) shows the fused result by Raman et al. for House sequence. The result exhibit color distortion and contrast loss. In contrast, proposed method produces result with vivid color quality and better contrast.
与 Raman 等人[20]的对比：图 3(f)展示了 Raman 等人对 House 序列的融合结果。该结果存在色彩失真和对比度损失的问题。相比之下，本文提出的方法能生成色彩生动且对比度更优的结果。

After examining the results by both subjective and objective evaluations, we observed that our method is able to faithfully reproduce all the features in the input pair. We also notice that the results obtained by DeepFuse are free of artifacts such as darker regions and mismatched colors. Our approach preserves the finer image details along with higher contrast and vivid colors. The quantitative comparison between proposed method and existing approaches in Table 2 also shows that proposed method outperforms others in most of the test sequences. From the execution times shown in Table 3 we can observe that our method is roughly 3-4$\times$ faster than Mertens et al. DeepFuse can be easily extended to more input images by adding additional streams before merge layer. We have trained DeepFuse for sequences with 3 and 4 images. For sequences with 3 images, average MEF SSIM score for DF is 0.987 and 0.979 for Mertens et al. For sequences with 4 images, average MEF SSIM score for DF is 0.972 and 0.978 for Mertens et al. For sequences with 4 images, we attribute dip in performance to insufficient training data. With more training data, DF can be trained to perform better in such cases as well.
通过主客观评估检验结果后，我们发现本方法能准确复现输入图像对的所有特征。同时观察到 DeepFuse 生成的结果不存在暗区或色彩失配等伪影问题。该方法在保持更高对比度与鲜艳色彩的同时，还能保留更精细的图像细节。表 2 所示定量对比表明，在多数测试序列中本方法优于现有方案。由表 3 执行时间可见，本方法速度较 Mertens 等人方案快约 3-4 倍。通过合并层前增加额外数据流，DeepFuse 可轻松扩展至多图像输入。我们已针对 3 幅和 4 幅图像序列训练模型：对于 3 幅图像序列，DF 的平均 MEF-SSIM 得分为 0.987，Mertens 方案为 0.979；4 幅图像序列中 DF 得分为 0.972，Mertens 方案为 0.978。我们认为 4 幅序列性能下降源于训练数据不足。 随着训练数据的增加，DF 模型也能通过训练在这些情况下表现更优。

In this section, we discuss the possibility of applying our DeepFuse model for solving other image fusion problems. Due to the limited depth-of-field in the present day cameras, only object in limited range of depth are focused and the remaining regions appear blurry. In such scenario, Multi-Focus Fusion (MFF) techniques are used to fuse images taken with varying focus to generate a single all-in-focus image. MFF problem is very similar to MEF, except that the input images have varying focus than varying exposure for MEF. To test the generalizability of CNN, we have used the already trained DeepFuse CNN to fuse multi-focus images without any fine-tuning for MFF problem. Fig. 9 shows that the DeepFuse results on publicly available multi-focus dataset show that the filters of CNN have learnt to identify proper regions in each input image and successfully fuse them together. It can also be seen that the learnt CNN filters are generic and could be applied for general image fusion.
本节探讨将 DeepFuse 模型应用于解决其他图像融合问题的可能性。由于现代相机景深有限，只有特定深度范围内的物体能清晰对焦，其余区域则呈现模糊状态。在此情况下，多焦点融合技术通过整合不同对焦位置的图像，生成单张全清晰图像。多焦点融合问题与多曝光融合高度相似，区别仅在于输入图像的变化因素是对焦位置而非曝光程度。为测试卷积神经网络的泛化能力，我们直接使用已训练的 DeepFuse CNN 模型进行多焦点图像融合，未针对该问题做任何微调。图 9 显示，在公开多焦点数据集上的融合结果表明，CNN 滤波器已学会识别各输入图像的恰当区域并成功实现融合。这同时证明所学得的 CNN 滤波器具有通用性，可推广至一般图像融合任务。