上海交通大学学报

上海交通大学学报 >

2024 , Vol. 58 >Issue 10: 1606 - 1617

DOI: https://doi.org/10.16183/j.cnki.jsjtu.2023.043

电子信息与电气工程

面向多天气退化图像恢复的自注意力扩散模型

  • 秦菁 ,
  • 文渊博 ,
  • 高涛 ,
  • 刘瑶
展开
  • 长安大学 信息工程学院,西安 710064
秦 菁(1975—),讲师,现主要从事信号处理及图像处理研究.
文渊博,博士生;E-mail:wyb@chd.edu.cn.

收稿日期: 2023-02-10

  修回日期: 2023-03-02

  录用日期: 2023-03-09

  网络出版日期: 2023-03-22

基金资助

国家国家自然科学基金(52172379);长安大学中央高校基本科研业务费专项资金(300102242901)

收起

A Transformer-Based Diffusion Model for All-in-One Weather-Degraded Image Restoration

  • QIN Jing ,
  • WEN Yuanbo ,
  • GAO Tao ,
  • LIU Yao
Expand
  • School of Information and Engineering, Chang’an University, Xi’an 710064, China

Received date: 2023-02-10

  Revised date: 2023-03-02

  Accepted date: 2023-03-09

  Online published: 2023-03-22

Fold

摘要

复杂天气下的图像恢复对后续高级计算机视觉任务具有重要意义.然而,多数现有图像恢复算法仅能去除单一天气退化,鲜有针对多天气退化图像恢复的同一模型.对此,结合去噪扩散概率模型和视觉Transformer,提出一种用于多天气退化图像恢复的自注意力扩散模型.首先,利用天气退化图像作为条件来引导扩散模型反向采样生成去除退化的干净背景图像.其次,提出次空间转置自注意力噪声估计网络,利用退化图像和噪化状态来估计噪声分布,包括次空间转置自注意力机制 (STSA) 和双分组门控前馈网络 (DGGFFN).STSA利用次空间变换系数实现有效学习特征全局性长距离依赖的同时,可显著降低计算负担;DGGFFN利用双分组门控机制来增强前馈网络的非线性表征能力.实验结果表明,在5个天气退化图像数据集上,相比近来同类算法All-in-One和TransWeather,本文算法所得恢复图像的平均峰值信噪比分别提高3.68和3.08 dB,平均结构相似性分别提高2.93%和3.13%.

关键词： 计算机视觉; 扩散模型; 图像恢复; Transformer; 天气退化图像

本文引用格式

秦菁 , 文渊博 , 高涛 , 刘瑶 . 面向多天气退化图像恢复的自注意力扩散模型[J]. 上海交通大学学报, 2024 , 58(10) : 1606 -1617 . DOI: 10.16183/j.cnki.jsjtu.2023.043

Abstract

Image restoration under adverse weather conditions is of great significance for the subsequent advanced computer vision tasks. However, most existing image restoration algorithms only remove single weather degradation, and few studies has been conducted on all-in-one weather-degraded image restoration. The denoising diffusion probability model is combined with Vision Transformer to propose a Transformer-based diffusion model for all-in-one weather-degraded image restoration. First, the weather-degraded image is utilized as the condition to guide the reverse sampling of diffusion model and generate corresponding clean background image. Then, the subspace transposed Transformer for noise estimation (NE-STT) is proposed, which utilizes the degraded image and the noisy state to estimate noise distribution, including the subspace transposed self-attention (STSA) mechanism and a dual grouped gated feed-forward network (DGGFFN). The STSA adopts subspace transformation coefficient to effectively capture global long-range dependencies while significantly reducing computational burden. The DGGFFN employs the dual grouped gated mechanism to enhance the nonlinear characterization ability of feed-forward network. The experimental results show that in comparison with the recently developed algorithms, such as All-in-One and TransWeather, the method proposed obtains a performance gain of 3.68 and 3.08 dB in average peak signal-to-noise ratio while 2.93% and 3.13% in average structural similarity on 5 weather-degraded datasets.

Key words： computer vision; diffusion model; image restoration; Transformer; weather-degraded image

参考文献

[1] 高涛, 文渊博, 陈婷, 等. 基于窗口自注意力网络的单图像去雨算法[J]. 上海交通大学学报, 2023, 57(5): 613-623.
  GAO Tao, WEN Yuanbo, CHEN Ting, et al. A single image deraining algorithm based on Swin Transformer[J]. Journal of Shanghai Jiao Tong University, 2023, 57(5): 613-623.
[2] 黄鹤, 胡凯益, 李战一, 等. 融合MCAP和GRTV正则化的无人机航拍建筑物图像去雾方法[J]. 上海交通大学学报, 2023, 57(3): 613-623.
  HUANG He, HU Kaiyi, LI Zhanyi, et al. An image dehazing method for UAV aerial photography to buildings combining MCAP and GRTV regularization[J]. Journal of Shanghai Jiao Tong University, 2023, 57(3): 613-623.
[3] LI R, ROBBY T T, LOONG-FAH C. All in one bad weather removal using architectural search[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 3175-3185.
[4] VALANARASU J M J, YASARLA R, PATEL V M. Transweather: Transformer-based restoration of images degraded by adverse weather conditions[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 2353-2363.
[5] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[6] KINGMA D P, WELLING M. Auto-encoding variational bayes[DB/OL]. (2013-12-20)[2023-02-06]. https://arxiv.org/abs/1312.6114.
[7] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
[8] DHARIWAL P, NICHOL A. Diffusion models beat gans on image synthesis[J]. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.
[9] PEEBLES W, XIE S. Scalable diffusion models with Transformers[DB/OL]. (2022-12-19)[2023-02-06]. https://arxiv.org/abs/2212.09748.
[10] WANG Z, CUN X, BAO J, et al. Uformer: A general u-shaped transformer for image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 17683-17693.
[11] ZAMIR S W, ARORA A, KHAN S, et al. Restormer: Efficient transformer for high-resolution image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 5728-5739.
[12] YAO T, LI Y, PAN Y, et al. Dual vision transformer[DB/OL]. (2022-07-11) [2023-02-06]. https://arxiv.org/abs/2207.04976.
[13] CHEN L, CHU X, ZHANG X, et al. Simple baselines for image restoration[C]// Proceedings of the European Conference on Computer Vision. Tel Aviv, Israel: Springer, 2022: 17-33.
[14] LIU Y F, JAW D W, HUANG S C, et al. DesnowNet: Context-aware deep network for snow removal[J]. IEEE Transactions on Image Processing, 2018, 27(6): 3064-3073.
[15] 鲍先富, 强赞霞, 杨关. 功能解耦和谱特征融合的雪霾消除模型[J]. 计算机工程与应用, 2023, 59(13): 211-219.
  BAO Xianfu, QIANG Zanxia, YANG Guan. Generative adverbial network for function decoupling and edge feature fusion for snow and haze elimination[J]. Computer Engineering & Applications, 2023, 59(13):211-219.
[16] 柴国强, 王大为, 芦宾, 等. 基于注意机制的轻量化稠密连接网络单幅图像去雨[J]. 北京航空航天大学学报, 2022, 48(11): 2186-2192.
  CHAI Guoqiang, WANG Dawei, LU Bin, et al. Lightweight densely connected network based on attention mechanism for single-image deraining[J]. Journal of Beijing University of Aeronautics & Astronautics, 2022, 48(11): 2186-2192.
[17] QIAN R, TAN R T, YANG W, et al. Attentive generative adversarial network for raindrop removal from a single image[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2482-2491.
[18] CHEN H, WANG Y, GUO T, et al. Pre-trained image processing transformer[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 12299-12310.
[19] LI B, LIU X, HU P, et al. All-in-one image restoration for unknown corruption[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 17452-17462.
[20] LI R, CHEONG L F, TAN R T. Heavy rain image restoration: Integrating physics model and conditional adversarial learning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 1633-1642.
[21] LI S, ARAUJO I B, REN W, et al. Single image deraining: A comprehensive benchmark analysis[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 3838-3847.
[22] LI X, WU J, LIN Z, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]// Proceedings of the European Conference on Computer Vision. Salty Lake City, UT, USA: Springer, 2018: 254-269.
[23] WANG T, YANG X, XU K, et al. Spatial attentive single-image deraining with a high quality real rain dataset[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 12270-12279.
[24] CHEN W, FANG H, DING J, et al. JSTASR: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal[C]// Proceedings of the European Conference on Computer Vision. Glasgow, UK: Springer, 2020: 754-770.
[25] LIANG J, CAO J, SUN G, et al. Swinir: Image restoration using swin transformer[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, canada: IEEE, 2021: 1833-1844.
[26] ZHANG K, LI R, YU Y, et al. Deep dense multi-scale network for snow removal using semantic and depth priors[J]. IEEE Transactions on Image Processing, 2021, 30: 7419-7431.
[27] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2223-2232.
[28] ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Venice, Italy: IEEE, 2017: 1125-1134.
[29] JIANG K, WANG Z, YI P, et al. Rain-free and residue hand-in-hand: A progressive coupled network for real-time image deraining[J]. IEEE Transactions on Image Processing, 2021, 30: 7404-7418.
[30] ZAMIR S W, ARORA A, KHAN S, et al. Multi-stage progressive image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 14821-14831.
[31] LIU X, SUGANUMA M, SUN Z, et al. Dual residual networks leveraging the potential of paired operations for image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 7007-7016.
[32] QUAN Y, DENG S, CHEN Y, et al. Deep learning for seeing through window with raindrops[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul Korea: IEEE, 2019: 2463-2471.
[33] QUAN R, YU X, LIANG Y, et al. Removing raindrops and rain streaks in one go[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 9147-9156.
[34] XIAO J, FU X, LIU A, et al. Image de-raining transformer[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2022: 1-18.
Options
文章导航

/

〈 〉