Stagewise Training for Hybrid-Distorted Image Restoration

doi:10.1007/s12204-022-2453-2

Abstract

Abstract: Image restoration is the problem of restoring a real degraded image. Previous studies mostly focused on single distortion. However, most of the real images experience multiple distortions, and single distortion image restoration algorithms can not effectively improve the image quality. Moreover, few existing hybrid distortion image restoration algorithms can not deal with single distortion. Therefore, an end-to-end pipeline network based on stagewise training is proposed in this paper. Specifically, the network selects three typical image restoration tasks: denoising, inpainting, and super resolution. The whole training process is divided into single distortion training, hybrid distortion training of two types, and hybrid distortion training of three types. The design of loss function draws on the idea of deep supervision. Experimental results prove that the proposed method is not only superior to other methods in hybrid-distorted image restoration, but also suitable for single distortion image restoration.

Key words: image restoration, stagewise training, hybrid distortion, single distortion

摘要： 图像复原是将退化图像恢复至接近理想图像的过程。以前的研究大多集中在单一失真图像上，然而大多数真实图像都经历了多种失真，单一失真图像复原算法无法有效提高图像质量。此外，现有的几种混合失真图像复原算法不具备处理单一失真的兼容性。因此，本文提出了一种基于分阶段训练的端到端神经网络。具体来说，该网络选择了三个典型的图像复原任务：图像去噪、图像修复和图像超分辨率。整个训练过程分为单一失真训练、两种类型的混合失真训练和三种类型的混合失真训练。损失函数的设计是基于深度监督的思想。实验结果表明，该方法不仅在混合失真图像复原方面优于其他方法，而且适用于单一失真图像复原。

关键词: 图像复原，分阶段训练，混合失真，单一失真

CLC Number:

TP391.9

HOU Shujuan* (侯舒娟)，ZHU Wenping (朱文萍)，LI Hai (李海). Stagewise Training for Hybrid-Distorted Image Restoration[J]. J Shanghai Jiaotong Univ Sci, 2023, 28(6): 793-801.

References 20

[10]	LIU H Y, JIANG B, XIAO Y, et al. Coherent semantic attention for image inpainting [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019: 4169-4178.
[1]	FU X Y, HUANG J B, ZENG D L, et al. Removing rain from single images via a deep detail network [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 1715-1723.
[11]	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2472-2481.
[2]	YANG W H, TAN R T, FENG J S, et al. Deep joint rain detection and removal from a single image [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 1685-1694.
[12]	DAI T, CAI J R, ZHANG Y B, et al. Second-order attention network for single image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 11057-11066.
[3]	QIAN R, TAN R T, YANG W H, et al. Attentive generative adversarial network for raindrop removal from A single image [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2482-2491.
[13]	ZHANG K, VAN GOOL L, TIMOFTE R. Deep unfolding network for image super-resolution [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 3214-3223.
[4]	JIN X, CHEN Z B, LIN J X, et al. Unsupervised single image deraining with self-supervised constraints [C]//2019 IEEE International Conference on Image Processing. Taipei, China: IEEE, 2019: 2761-2765.
[14]	YU K, DONG C, LIN L, et al. Crafting a toolchain for image restoration by deep reinforcement learning [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2443-2452.
[5]	ZHANG K, ZUO W, ZHANG L. FFDNet: toward a fast and flexible solution for CNN based image denoising [J]. IEEE Transactions on Image Processing, 2018: 27(9): 4608-4622.
[15]	SUGANUMA M, LIU X, OKATANI T. Attentionbased adaptive selection of operations for image restoration in the presence of unknown combined distortions [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 9031-9040.
[6]	GUO S, YAN Z F, ZHANG K, et al. Toward convolutional blind denoising of real photographs [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 1712-1722.
[16]	QIAN G C, WANG Y H, DONG C, et al. Rethinking the pipeline of demosaicing, denoising and super-resolution [EB/OL]. (2019-05-07). https://arxiv.org/abs/1905.02538.
[7]	TIAN C W, XU Y, ZUO W M. Image denoising using deep CNN with batch renormalization [J]. Neural Networks, 2020, 121: 461-473.
[17]	WANG L, KIM T K, YOON K J. EventSR: from asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 8312-8322.
[8]	DONG L F, GAN Y Z, MAO X L, et al. Learning deep representations using convolutional auto-encoders with symmetric skip connections [C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary, AB, Canada: IEEE, 2018: 3006- 3010.
[18]	YANG T B, YAN Y, YUAN Z N, et al. Why does stagewise training accelerate convergence of testing error over SGD? [EB/OL]. (2018-12-10). https://arxiv.org/abs/1812.03934.
[9]	WANG N, LI J Y, ZHANG L F, et al. MUSICAL: Multi-scale image contextual attention learning for inpainting [C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao, China: International Joint Conferences on Artificial Intelligence Organization, 2019: 3748-3754.
[19]	ZHOU Y Y, XIE L X, FISHMAN E K, et al. Deep supervision for pancreatic cyst segmentation in abdominal CT scans [M]//Medical image computing and computer assisted intervention - MICCAI 2017. Cham: Springer, 2017: 222-230.
[10]	LIU H Y, JIANG B, XIAO Y, et al. Coherent semantic attention for image inpainting [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019: 4169-4178.
[20]	LEMPITSKY V, VEDALDI A, ULYANOV D. Deep image prior [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 9446-9454.
[11]	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2472-2481.
[12]	DAI T, CAI J R, ZHANG Y B, et al. Second-order attention network for single image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 11057-11066.
[13]	ZHANG K, VAN GOOL L, TIMOFTE R. Deep unfolding network for image super-resolution [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 3214-3223.
[14]	YU K, DONG C, LIN L, et al. Crafting a toolchain for image restoration by deep reinforcement learning [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2443-2452.
[15]	SUGANUMA M, LIU X, OKATANI T. Attentionbased adaptive selection of operations for image restoration in the presence of unknown combined distortions [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 9031-9040.
[16]	QIAN G C, WANG Y H, DONG C, et al. Rethinking the pipeline of demosaicing, denoising and super-resolution [EB/OL]. (2019-05-07). https://arxiv.org/abs/1905.02538.
[17]	WANG L, KIM T K, YOON K J. EventSR: from asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 8312-8322.
[18]	YANG T B, YAN Y, YUAN Z N, et al. Why does stagewise training accelerate convergence of testing error over SGD? [EB/OL]. (2018-12-10). https://arxiv.org/abs/1812.03934.
[19]	ZHOU Y Y, XIE L X, FISHMAN E K, et al. Deep supervision for pancreatic cyst segmentation in abdominal CT scans [M]//Medical image computing and computer assisted intervention - MICCAI 2017. Cham: Springer, 2017: 222-230.
[20]	LEMPITSKY V, VEDALDI A, ULYANOV D. Deep image prior [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 9446-9454.