Image restoration is the problem of restoring a real degraded image. Previous studies mostly focused on single distortion. However, most of the real images experience multiple distortions, and single distortion image restoration algorithms can not effectively improve the image quality. Moreover, few existing hybrid distortion image restoration algorithms can not deal with single distortion. Therefore, an end-to-end pipeline network based on stagewise training is proposed in this paper. Specifically, the network selects three typical image restoration tasks: denoising, inpainting, and super resolution. The whole training process is divided into single distortion training, hybrid distortion training of two types, and hybrid distortion training of three types. The design of loss function draws on the idea of deep supervision. Experimental results prove that the proposed method is not only superior to other methods in hybrid-distorted image restoration, but also suitable for single distortion image restoration.
HOU Shujuan* (侯舒娟)，ZHU Wenping (朱文萍)，LI Hai (李海)
. Stagewise Training for Hybrid-Distorted Image Restoration[J]. Journal of Shanghai Jiaotong University(Science), 2023
, 28(6)
: 793
-801
.
DOI: 10.1007/s12204-022-2453-2
[1] FU X Y, HUANG J B, ZENG D L, et al. Removing rain from single images via a deep detail network [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 1715-1723.
[2] YANG W H, TAN R T, FENG J S, et al. Deep joint rain detection and removal from a single image [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 1685-1694.
[3] QIAN R, TAN R T, YANG W H, et al. Attentive generative adversarial network for raindrop removal from A single image [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2482-2491.
[4] JIN X, CHEN Z B, LIN J X, et al. Unsupervised single image deraining with self-supervised constraints [C]//2019 IEEE International Conference on Image Processing. Taipei, China: IEEE, 2019: 2761-2765.
[5] ZHANG K, ZUO W, ZHANG L. FFDNet: toward a fast and flexible solution for CNN based image denoising [J]. IEEE Transactions on Image Processing, 2018: 27(9): 4608-4622.
[6] GUO S, YAN Z F, ZHANG K, et al. Toward convolutional blind denoising of real photographs [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 1712-1722.
[7] TIAN C W, XU Y, ZUO W M. Image denoising using deep CNN with batch renormalization [J]. Neural Networks, 2020, 121: 461-473.
[8] DONG L F, GAN Y Z, MAO X L, et al. Learning deep representations using convolutional auto-encoders with symmetric skip connections [C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary, AB, Canada: IEEE, 2018: 3006- 3010.
[9] WANG N, LI J Y, ZHANG L F, et al. MUSICAL: Multi-scale image contextual attention learning for inpainting [C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao, China: International Joint Conferences on Artificial Intelligence Organization, 2019: 3748-3754.
[10] LIU H Y, JIANG B, XIAO Y, et al. Coherent semantic attention for image inpainting [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019: 4169-4178.
[11] ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2472-2481.
[12] DAI T, CAI J R, ZHANG Y B, et al. Second-order attention network for single image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 11057-11066.
[13] ZHANG K, VAN GOOL L, TIMOFTE R. Deep unfolding network for image super-resolution [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 3214-3223.
[14] YU K, DONG C, LIN L, et al. Crafting a toolchain for image restoration by deep reinforcement learning [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 2443-2452.
[15] SUGANUMA M, LIU X, OKATANI T. Attentionbased adaptive selection of operations for image restoration in the presence of unknown combined distortions [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 9031-9040.
[16] QIAN G C, WANG Y H, DONG C, et al. Rethinking the pipeline of demosaicing, denoising and super-resolution [EB/OL]. (2019-05-07). https://arxiv.org/abs/1905.02538.
[17] WANG L, KIM T K, YOON K J. EventSR: from asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 8312-8322.
[18] YANG T B, YAN Y, YUAN Z N, et al. Why does stagewise training accelerate convergence of testing error over SGD? [EB/OL]. (2018-12-10). https://arxiv.org/abs/1812.03934.
[19] ZHOU Y Y, XIE L X, FISHMAN E K, et al. Deep supervision for pancreatic cyst segmentation in abdominal CT scans [M]//Medical image computing and computer assisted intervention - MICCAI 2017. Cham: Springer, 2017: 222-230.
[20] LEMPITSKY V, VEDALDI A, ULYANOV D. Deep image prior [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 9446-9454.
