Journal of Shanghai Jiao Tong University

Journal of Shanghai Jiaotong University >

2024 , Vol. 58 >Issue 10: 1606 - 1617

DOI: https://doi.org/10.16183/j.cnki.jsjtu.2023.043

Original article

A Transformer-Based Diffusion Model for All-in-One Weather-Degraded Image Restoration

  • QIN Jing ,
  • WEN Yuanbo ,
  • GAO Tao ,
  • LIU Yao
Expand
  • School of Information and Engineering, Chang’an University, Xi’an 710064, China

Received date: 2023-02-10

  Revised date: 2023-03-02

  Accepted date: 2023-03-09

  Online published: 2023-03-22

Fold

Abstract

Image restoration under adverse weather conditions is of great significance for the subsequent advanced computer vision tasks. However, most existing image restoration algorithms only remove single weather degradation, and few studies has been conducted on all-in-one weather-degraded image restoration. The denoising diffusion probability model is combined with Vision Transformer to propose a Transformer-based diffusion model for all-in-one weather-degraded image restoration. First, the weather-degraded image is utilized as the condition to guide the reverse sampling of diffusion model and generate corresponding clean background image. Then, the subspace transposed Transformer for noise estimation (NE-STT) is proposed, which utilizes the degraded image and the noisy state to estimate noise distribution, including the subspace transposed self-attention (STSA) mechanism and a dual grouped gated feed-forward network (DGGFFN). The STSA adopts subspace transformation coefficient to effectively capture global long-range dependencies while significantly reducing computational burden. The DGGFFN employs the dual grouped gated mechanism to enhance the nonlinear characterization ability of feed-forward network. The experimental results show that in comparison with the recently developed algorithms, such as All-in-One and TransWeather, the method proposed obtains a performance gain of 3.68 and 3.08 dB in average peak signal-to-noise ratio while 2.93% and 3.13% in average structural similarity on 5 weather-degraded datasets.

Key words： computer vision; diffusion model; image restoration; Transformer; weather-degraded image

Cite this article

QIN Jing , WEN Yuanbo , GAO Tao , LIU Yao . A Transformer-Based Diffusion Model for All-in-One Weather-Degraded Image Restoration[J]. Journal of Shanghai Jiaotong University, 2024 , 58(10) : 1606 -1617 . DOI: 10.16183/j.cnki.jsjtu.2023.043

References

[1] 高涛, 文渊博, 陈婷, 等. 基于窗口自注意力网络的单图像去雨算法[J]. 上海交通大学学报, 2023, 57(5): 613-623.
  GAO Tao, WEN Yuanbo, CHEN Ting, et al. A single image deraining algorithm based on Swin Transformer[J]. Journal of Shanghai Jiao Tong University, 2023, 57(5): 613-623.
[2] 黄鹤, 胡凯益, 李战一, 等. 融合MCAP和GRTV正则化的无人机航拍建筑物图像去雾方法[J]. 上海交通大学学报, 2023, 57(3): 613-623.
  HUANG He, HU Kaiyi, LI Zhanyi, et al. An image dehazing method for UAV aerial photography to buildings combining MCAP and GRTV regularization[J]. Journal of Shanghai Jiao Tong University, 2023, 57(3): 613-623.
[3] LI R, ROBBY T T, LOONG-FAH C. All in one bad weather removal using architectural search[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 3175-3185.
[4] VALANARASU J M J, YASARLA R, PATEL V M. Transweather: Transformer-based restoration of images degraded by adverse weather conditions[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 2353-2363.
[5] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[6] KINGMA D P, WELLING M. Auto-encoding variational bayes[DB/OL]. (2013-12-20)[2023-02-06]. https://arxiv.org/abs/1312.6114.
[7] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
[8] DHARIWAL P, NICHOL A. Diffusion models beat gans on image synthesis[J]. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.
[9] PEEBLES W, XIE S. Scalable diffusion models with Transformers[DB/OL]. (2022-12-19)[2023-02-06]. https://arxiv.org/abs/2212.09748.
[10] WANG Z, CUN X, BAO J, et al. Uformer: A general u-shaped transformer for image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 17683-17693.
[11] ZAMIR S W, ARORA A, KHAN S, et al. Restormer: Efficient transformer for high-resolution image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 5728-5739.
[12] YAO T, LI Y, PAN Y, et al. Dual vision transformer[DB/OL]. (2022-07-11) [2023-02-06]. https://arxiv.org/abs/2207.04976.
[13] CHEN L, CHU X, ZHANG X, et al. Simple baselines for image restoration[C]// Proceedings of the European Conference on Computer Vision. Tel Aviv, Israel: Springer, 2022: 17-33.
[14] LIU Y F, JAW D W, HUANG S C, et al. DesnowNet: Context-aware deep network for snow removal[J]. IEEE Transactions on Image Processing, 2018, 27(6): 3064-3073.
[15] 鲍先富, 强赞霞, 杨关. 功能解耦和谱特征融合的雪霾消除模型[J]. 计算机工程与应用, 2023, 59(13): 211-219.
  BAO Xianfu, QIANG Zanxia, YANG Guan. Generative adverbial network for function decoupling and edge feature fusion for snow and haze elimination[J]. Computer Engineering & Applications, 2023, 59(13):211-219.
[16] 柴国强, 王大为, 芦宾, 等. 基于注意机制的轻量化稠密连接网络单幅图像去雨[J]. 北京航空航天大学学报, 2022, 48(11): 2186-2192.
  CHAI Guoqiang, WANG Dawei, LU Bin, et al. Lightweight densely connected network based on attention mechanism for single-image deraining[J]. Journal of Beijing University of Aeronautics & Astronautics, 2022, 48(11): 2186-2192.
[17] QIAN R, TAN R T, YANG W, et al. Attentive generative adversarial network for raindrop removal from a single image[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2482-2491.
[18] CHEN H, WANG Y, GUO T, et al. Pre-trained image processing transformer[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 12299-12310.
[19] LI B, LIU X, HU P, et al. All-in-one image restoration for unknown corruption[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. NewOrleans, LA, USA: IEEE, 2022: 17452-17462.
[20] LI R, CHEONG L F, TAN R T. Heavy rain image restoration: Integrating physics model and conditional adversarial learning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 1633-1642.
[21] LI S, ARAUJO I B, REN W, et al. Single image deraining: A comprehensive benchmark analysis[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 3838-3847.
[22] LI X, WU J, LIN Z, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]// Proceedings of the European Conference on Computer Vision. Salty Lake City, UT, USA: Springer, 2018: 254-269.
[23] WANG T, YANG X, XU K, et al. Spatial attentive single-image deraining with a high quality real rain dataset[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 12270-12279.
[24] CHEN W, FANG H, DING J, et al. JSTASR: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal[C]// Proceedings of the European Conference on Computer Vision. Glasgow, UK: Springer, 2020: 754-770.
[25] LIANG J, CAO J, SUN G, et al. Swinir: Image restoration using swin transformer[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, canada: IEEE, 2021: 1833-1844.
[26] ZHANG K, LI R, YU Y, et al. Deep dense multi-scale network for snow removal using semantic and depth priors[J]. IEEE Transactions on Image Processing, 2021, 30: 7419-7431.
[27] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2223-2232.
[28] ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Venice, Italy: IEEE, 2017: 1125-1134.
[29] JIANG K, WANG Z, YI P, et al. Rain-free and residue hand-in-hand: A progressive coupled network for real-time image deraining[J]. IEEE Transactions on Image Processing, 2021, 30: 7404-7418.
[30] ZAMIR S W, ARORA A, KHAN S, et al. Multi-stage progressive image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 14821-14831.
[31] LIU X, SUGANUMA M, SUN Z, et al. Dual residual networks leveraging the potential of paired operations for image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 7007-7016.
[32] QUAN Y, DENG S, CHEN Y, et al. Deep learning for seeing through window with raindrops[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul Korea: IEEE, 2019: 2463-2471.
[33] QUAN R, YU X, LIANG Y, et al. Removing raindrops and rain streaks in one go[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 9147-9156.
[34] XIAO J, FU X, LIU A, et al. Image de-raining transformer[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2022: 1-18.
Options
Outlines

/

〈 〉