基于窗口自注意力网络的单图像去雨算法

doi:10.16183/j.cnki.jsjtu.2022.032

上海交通大学学报 ›› 2023, Vol. 57 ›› Issue (5): 613-623.doi: 10.16183/j.cnki.jsjtu.2022.032

所属专题：《上海交通大学学报》2023年“电子信息与电气工程”专题

基于窗口自注意力网络的单图像去雨算法

高涛¹, 文渊博¹(), 陈婷¹, 张静²

1.长安大学信息工程学院,西安 710064
2.澳大利亚国立大学工程与计算机学院,澳大利亚堪培拉 2600

收稿日期:2022-02-14 修回日期:2022-03-20 接受日期:2022-04-28 出版日期:2023-05-28 发布日期:2023-06-02
通讯作者: 文渊博 E-mail:wyb@chd.edu.cn.
作者简介:高涛(1981-),教授,博士生导师,现主要从事数字图像处理和模式识别研究.
基金资助:
国家重点研发计划项目(2019YFE0108300);国家自然科学基金项目(52172379);国家自然科学基金项目(62001058);陕西省重点研发计划(2019GY-039);中央高校基本科研业务费专项资金项目(300102242901);中央高校基本科研业务费专项资金项目(300102112601)

A Single Image Deraining Algorithm Based on Swin Transformer

GAO Tao¹, WEN Yuanbo¹(), CHEN Ting¹, ZHANG Jing²

1. School of Information Engineering, Chang’an University, Xi’an 710064, China
2. College of Engineering and Computer Science, Australian National University, Canberra 2600, ACT, Australia

Received:2022-02-14 Revised:2022-03-20 Accepted:2022-04-28 Online:2023-05-28 Published:2023-06-02
Contact: WEN Yuanbo E-mail:wyb@chd.edu.cn.

摘要/Abstract

摘要：

单图像去雨研究旨在利用退化的雨图恢复出无雨图像,而现有的基于深度学习的去雨算法未能有效地利用雨图的全局性信息,导致去雨后的图像损失部分细节和结构信息.针对此问题,提出一种基于窗口自注意力网络 (Swin Transformer) 的单图像去雨算法.该算法网络主要包括浅层特征提取模块和深度特征提取网络两部分.前者利用上下文信息聚合输入来适应雨痕分布的多样性,进而提取雨图的浅层特征.后者利用Swin Transformer捕获全局性信息和像素点间的长距离依赖关系,并结合残差卷积和密集连接强化特征学习,最后通过全局残差卷积输出去雨图像.此外,提出一种同时约束图像边缘和区域相似性的综合损失函数来进一步提高去雨图像的质量.实验表明,与目前单图像去雨表现优秀的算法MSPFN、 MPRNet相比,该算法使去雨图像的峰值信噪比提高0.19 dB和2.17 dB,结构相似性提高3.433%和1.412%,同时网络模型参数量下降84.59%和34.53%,前向传播平均耗时减少21.25%和26.67%.

关键词: 计算机视觉, 单图像去雨, 窗口自注意力网络, 残差网络, 自注意力机制, 空洞卷积

Abstract:

Single image deraining aims to recover the rain-free image from rainy image. Most existing deraining methods based on deep learning do not utilize the global information of rainy image effectively, which makes them lose much detailed and structural information after processing. Focusing on this issue, this paper proposes a single image deraining algorithm based on Swin Transformer. The network mainly includes a shallow features extraction module and a deep features extraction network. The former exploits the context information aggregation module to adapt to the distribution diversity of rain streaks and extracts the shallow features of rainy image. The latter uses Swin Transformer to capture the global information and long-distance dependencies between different pixels, in combination with residual convolution and dense connection to strengthen features learning. Finally, the derained image is obtained through a global residual convolution. In addition, this paper proposes a novel comprehensive loss function that constrains the similarity of image edges and regions synchronously to further improve the quality of derained image. Extensive experimental results show that, compared with the two state-of-the-art methods, MSPFN and MPRNet, the average peak signal-to-noise ratio of derained images of our method increases by 0.19 dB and 2.17 dB, and the average structural similarity increases by 3.433% and 1.412%. At the same time, the model parameters of the proposed network decreases by 84.59% and 34.53%, and the forward propagation time reduces by 21.25% and 26.67%.

Key words: computer vision, single image deraining, Swin Transformer, residual network, self-attention mechanism, dilated convolution

中图分类号:

高涛, 文渊博, 陈婷, 张静. 基于窗口自注意力网络的单图像去雨算法[J]. 上海交通大学学报, 2023, 57(5): 613-623.

GAO Tao, WEN Yuanbo, CHEN Ting, ZHANG Jing. A Single Image Deraining Algorithm Based on Swin Transformer[J]. Journal of Shanghai Jiao Tong University, 2023, 57(5): 613-623.

图/表 11

图1

图2

图3

表1

表2

表3

表4

图4

表5

图5

表6

参考文献 33

[1]	陈舒曼, 陈玮, 尹钟. 单幅图像去雨算法研究现状及展望[J]. 计算机应用研究, 2022, 39(1): 9-17.
	CHEN Shuman, CHEN Wei, YIN Zhong. Research status and prospect of single image rain removal algorithm[J]. Application Research of Computers, 2022, 39(1): 9-17.
[2]	DENG S, WEI M, WANG J, et al. Detail-recovery image deraining via context aggregation networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 14560-14569.
[3]	HE K, GKIOXARI G, DOLLÁR P, et al. Mask RCNN[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2961-2969.
[4]	王春波, 张卫东, 张文渊, 等. 复杂交通环境中车辆的视觉检测[J]. 上海交通大学学报, 2000, 34(12): 1680-1682.
	WANG Chunbo, ZHANG Weidong, ZHANG Wen-yuan, et al. Vision-based vehicles detection in complex traffic scenes[J]. Journal of Shanghai Jiao Tong University, 2000, 34(12): 1680-1682.
[5]	YANG W, TAN R T, WANG S, et al. Single image deraining: From model-based to data-driven and beyond[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(11): 4059-4077. doi: 10.1109/TPAMI.2020.2995190 URL
[6]	ZHENG X, LIAO Y, GUO W, et al. Single-image-based rain and snow removal using multi-guided filter[C]//International Conference on Neural Information Processing. Daegu, South Korea: APNNS, 2013: 258-265.
[7]	KANG L, LIN C, FU Y. Automatic single-image-based rain streaks removal via image decomposition[J]. IEEE Transactions on Image Processing, 2011, 21(4): 1742-1755. doi: 10.1109/TIP.2011.2179057 URL
[8]	LUO Y, XU Y, JI H. Removing rain from a single image via discriminative sparse coding[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 3397-3405.
[9]	LI Y, TAN R T, GUO X, et al. Rain streak removal using layer priors[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 2736-2744.
[10]	FU X, HUANG J, DING X, et al. Clearing the skies: A deep network architecture for single-image rain removal[J]. IEEE Transactions on Image Processing, 2017, 26(6): 2944-2956. doi: 10.1109/TIP.2017.2691802 pmid: 28410108
[11]	WEI W, MENG D, ZHAO Q, et al. Semi-supervised transfer learning for image rain removal[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 3877-3886.
[12]	ZHANG H, PATEL V M. Density-aware single image deraining using a multi-stream dense network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 695-704.
[13]	YASARLA R, PATEL V M. Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 8405-8414.
[14]	LI X, WU J, LIN Z, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]//Proceedings of the European Conference on Computer Vision. Salt Lake City, UT, USA: IEEE, 2018: 254-269.
[15]	REN D, ZUO W, HU Q, et al. Progressive image deraining networks: A better and simpler baseline[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 3937-3946.
[16]	JIANG K, WANG Z, YI P, et al. Multi-scale progressive fusion network for single image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE. 2020: 8346-8355.
[17]	ZAMIR S W, ARORA A, KHAN S, et al. Multi-stage progressive image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2021: 14821-14831.
[18]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. Long Beach, CA, USA: NIPS, 2017: 5998-6008.
[19]	LIU Z, LIN Y, CAO Y, et al. Swin Transformer: Hierarchical vision Transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 10012-10022.
[20]	XIAO T, DOLLAR P, SINGH M, et al. Early convolutions help transformers see better[C]//Thirty-Fifth Conference on Neural Information Processing Systems. Montreal, Canada: NIPS, 2021: 34.
[21]	YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[C]//International Conference on Leaning Representations. Caribe Hilton, San Juan, Puerto Rico: OpenReview. net, 2016: 1-13.
[22]	WANG P, CHEN P, YUAN Y, et al. Understanding convolution for semantic segmentation[C]//2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, NV, USA: IEEE, 2018: 1451-1460.
[23]	LIANG J, CAO J, SUN G, et al. SwinIR: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 1833-1844.
[24]	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 770-778.
[25]	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Venice, Italy: IEEE, 2017: 4700-4708.
[26]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[27]	KAMGAR-PARSI B, ROSENFELD A. Optimally isotropic Laplacian operator[J]. IEEE Transactions on Image Processing, 1999, 8(10): 1467-1472. doi: 10.1109/83.791975 URL
[28]	ZHANG H, SINDAGI V, PATEL V M. Image de-raining using a conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(11): 3943-3956. doi: 10.1109/TCSVT.76 URL
[29]	YANG W, TAN R T, FENG J, et al. Deep joint rain detection and removal from a single image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Venice, Italy: IEEE, 2017: 1357-1366.
[30]	FU X, LIANG B, HUANG Y, et al. Lightweight pyramid networks for image deraining[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 31(6): 1794-1807. doi: 10.1109/TNNLS.5962385 URL
[31]	HUYNH-THU Q, GHANBARI M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800-801. doi: 10.1049/el:20080522 URL
[32]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212. doi: 10.1109/LSP.2012.2227726 URL
[33]	LIU L, LIU B, HUANG H, et al. No-reference image quality assessment based on spatial and spectral entropies[J]. Signal Processing: Image Communication, 2014, 29(8): 856-863. doi: 10.1016/j.image.2014.06.006 URL

数据集	训练样本对	测试样本对	重命名
Rain14000^[10]	11 200	2 800	Test2800
Rain800^[28]	700	100	Test100
Rain100H^[29]	1 800	100	Rain100H
Rain100L^[29]	0	100	Rain100L
Rain1200^[12]	0	1 200	Test1200
Rain12^[9]	12	0	—

网络组成		组合方式
输入层	单卷积	√	√	√	√	√	×
	CGB	×	×	×	×	×	√
特征提取网络	ResNet	√	×	×	×	×	×
	STB	×	√	√	√	√	√
	RSTB	×	×	√	√	√	√
	DRSTB	×	×	×	√	√	√
	DRST	×	×	×	×	√	√
输出层	单卷积	√	√	√	√	√	√
PSNR/dB		25.41	27.35	28.94	30.27	32.15	34.83
SSIM		0.846	0.882	0.886	0.904	0.912	0.924

损失函数	PSNR/dB	SSIM
MSE	29.57	0.884
Edge	29.24	0.891
SSIM	30.68	0.903
(MSE, SSIM)	32.79	0.916
(Edge, SSIM)	34.83	0.924

算法	Test100^[28]		Rain100H^[29]		Rain100L^[29]		Test2800^[10]		Test1200^[12]		平均
算法	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB (G/%)	SSIM (G/%)
DerainNet^[10]	22.77	0.810	14.92	0.592	27.03	0.884	24.31	0.861	23.38	0.835	22.48 (45.6)↑	0.796 (17.3)↑
SEMI^[11]	22.35	0.788	16.56	0.486	25.03	0.842	24.43	0.782	26.05	0.822	22.88 (43.9)↑	0.744 (25.5)↑
DIDMDN^[12]	22.56	0.818	17.35	0.524	25.23	0.741	28.13	0.867	29.65	0.901	24.58 (34.0)↑	0.770 (21.3)↑
UMRL^[13]	24.41	0.829	26.01	0.832	29.18	0.923	29.97	0.905	30.55	0.910	28.02 (17.5)↑	0.880 (6.14)↑
RESCAN^[14]	25.00	0.835	26.36	0.786	29.80	0.881	31.29	0.904	30.51	0.882	28.59 (15.1)↑	0.857 (8.98)↑
PReNet^[15]	24.81	0.851	26.77	0.858	32.44	0.950	31.75	0.916	31.36	0.911	29.42 (11.9)↑	0.897 (4.12)↑
MSPFN^[16]	27.50	0.876	28.66	0.860	32.40	0.933	32.82	0.930	32.39	0.916	30.75 (7.06)↑	0.903 (3.43)↑
MPRNet^[17]	30.27	0.897	30.41	0.890	36.40	0.965	33.64	0.938	32.91	0.916	32.73 (0.58)↑	0.921 (1.41)↑
本文算法	28.28	0.913	30.22	0.904	37.53	0.979	33.76	0.952	34.83	0.924	32.92	0.934

算法	NIQE	SSEQ
RESCAN^[14]	5.066	24.18
PReNet^[15]	5.025	20.76
MSPFN^[16]	4.961	22.29
MPRNet^[17]	5.166	22.05
本文算法	4.946	18.93

基于窗口自注意力网络的单图像去雨算法

A Single Image Deraining Algorithm Based on Swin Transformer

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 33

相关文章 6

编辑推荐

Metrics

本文评价

算法	参数量×10^-6	浮点运算次数×10^-9	平均耗时/s
MSPFN^[16]	15.82	37.86	0.045
MPRNet^[17]	3.637	8.841	0.048
本文算法	2.381	9.688	0.033

[1]	曹现刚1, 2，雷卓1，李彦川1，张梦园1，段欣宇1. 基于Self-Attention-LSTM神经网络的设备剩余寿命预测方法[J]. J Shanghai Jiaotong Univ Sci, 2023, 28(5): 652-664.
[2]	张峻宁, 苏群星, 王成, 徐超, 李一宁. 一种改进变换网络的域自适应语义分割网络[J]. 上海交通大学学报, 2021, 55(9): 1158-1168.
[3]	武光利, 郭振洲, 李雷霆, 王成祥. 融合FCN和LSTM的视频异常事件检测[J]. 上海交通大学学报, 2021, 55(5): 607-614.
[4]	蒋兴浩, 赵泽宇, 许可. 基于视觉的飞行器智能目标检测对抗攻击技术[J]. 空天防御, 2021, 4(1): 8-13.
[5]	马波，周越. 一种新的多视角人脸跟踪算法[J]. 上海交通大学学报（自然版）, 2010, 44(07): 902-0906.
[6]	应俊豪,张秀彬. 矿石颗粒尺度分布的双圆算法[J]. 上海交通大学学报（自然版）, 2010, 44(03): 384-0388.