利用生成对抗网络实现水下图像增强

doi:10.16183/j.cnki.jsjtu.2021.075

摘要/Abstract

摘要：

提出一种基于生成对抗模型的水下图像修正与增强算法.该算法将多尺度内核应用于改进的残差模块中,以此构建生成器,实现多感受野特征信息的提取与融合;判别器设计考虑了全局信息与局部细节的关系,建立了全局-区域双判别结构,能够保证整体风格与边缘纹理的一致性;最后,根据人类视觉感官系统设计了无监督损失函数,此部分无需参考图像进行约束,同时其与对抗损失和内容损失一起进行联合优化,能够得到更优的色彩和结构表现.在多个数据集上进行实验分析表明,此算法能较好地修正色偏、对比度,保护细节信息不丢失,在主客观指标上都优于典型对比算法.

关键词: 水下图像增强, 生成对抗网络, 残差结构, 无监督学习

Abstract:

This paper proposes an underwater image correction and enhancement algorithm based on generative adversarial networks. In this algorithm, the multi-scale kernel is applied to the improved residual module to construct a generator, which realizes the extraction and fusion of multiple receptive fields feature information. The discriminator design considers the relationship between global information and local details, and establishes a global-region dual discriminator structure, which can ensure the consistency of overall style and edge texture. An unsupervised loss function based on human visual sensory system is proposed. Reference image constraints are not required, and the confrontation loss and the content loss are jointly optimized to obtain better color and structure performance. Experimental evaluations on multiple data sets show that the proposed algorithm can better correct color deviation and contrast, protect details from loss, and is superior to typical algorithms in subjective and objective indexes.

Key words: underwater image enhancement, generative adversarial networks, residual structure, unsupervised learning

中图分类号:

TP391.4

李钰, 杨道勇, 刘玲亚, 王易因. 利用生成对抗网络实现水下图像增强[J]. 上海交通大学学报, 2022, 56(2): 134-142.

LI Yu, YANG Daoyong, LIU Lingya, WANG Yiyin. Underwater Image Enhancement Based on Generative Adversarial Networks[J]. Journal of Shanghai Jiao Tong University, 2022, 56(2): 134-142.

图/表 12

图1

图2

图3

图4

图5

图6

表1

图7

表2

图8

表3

续表3

参考文献 28

[1]	WHITCOMB L, YOERGER D R, SINGH H, et al. Advances in underwater robot vehicles for deep ocean exploration: Navigation, control, and survey operations[M]//Robotics Research. New York, USA: Springer International Publishing, 2000: 439-448.
[2]	LU H M, LI Y J, SERIKAWA S. Computer vision for ocean observing[M]//Artificial Intelligence and Computer Vision. New York, USA: Springer International Publishing, 2016: 1-16.
[3]	GAUDRON J O, SURRE F, SUN T, et al. Long period grating-based optical fibre sensor for the underwater detection of acoustic waves[J]. Sensors and Actuators A: Physical, 2013, 201:289-293. doi: 10.1016/j.sna.2013.07.017 URL
[4]	YAN Z, MA J, TIAN J W, et al. A gravity gradient differential ratio method for underwater object detection[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(4):833-837. doi: 10.1109/LGRS.2013.2279485 URL
[5]	WANG Y, SONG W, FORTINO G, et al. An experimental-based review of image enhancement and image restoration methods for underwater imaging[J]. IEEE Access, 2019, 7:140233-140251. doi: 10.1109/Access.6287639 URL
[6]	YANG M, HU J T, LI C Y, et al. An in-depth survey of underwater image enhancement and restoration[J]. IEEE Access, 2019, 7:123638-123657. doi: 10.1109/Access.6287639 URL
[7]	LI C Y, GUO C L, REN W Q, et al. An underwater image enhancement benchmark dataset and beyond[J]. IEEE Transactions on Image Processing, 2020, 29:4376-4389. doi: 10.1109/TIP.83 URL
[8]	HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(12):2341-2353. doi: 10.1109/TPAMI.2010.168 URL
[9]	DREWS JR P, DO NASCIMENTO E, MORAES F, et al. Transmission estimation in underwater single images[C]// 2013 IEEE International Conference on Computer Vision Workshops. Sydney, NSW, Australia: IEEE, 2013: 825-830.
[10]	WANG Y, ZHANG J, CAO Y, et al. A deep CNN method for underwater image enhancement[C]// 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China: IEEE, 2017: 1382-1386.
[11]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11):139-144. doi: 10.1145/3422622 URL
[12]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 2242-2251.
[13]	ISLAM M J, XIA Y Y, SATTAR J. Fast underwater image enhancement for improved visual perception[J]. IEEE Robotics and Automation Letters, 2020, 5(2):3227-3234. doi: 10.1109/LSP.2016. URL
[14]	LI Y, YU W B, LIU L Y, et al. SRM-net: An effective end-to-end neural network for single image dehazing[C]// Proceedings of the 3rd International Conference on Video and Image Processing. New York, NY, USA: ACM, 2019: 74-78.
[15]	YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[EB/OL].(2016 -04-30)[2020-10-13].https://arxiv.org/abs/1511.07122.
[16]	IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4):1-14.
[17]	RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL].(2015 -11-15)[2020-10-13]. https://www.researchgate.net/publication/284476553_Unsupervised_Representation_Learning_with_Deep_Convolutional_Generative_Adversarial_Networks.
[18]	BARBOSA W V, AMARAL H G B, ROCHA T L, et al. Visual-quality-driven learning for underwater vision enhancement[C]// 2018 25th IEEE International Conference on Image Processing (ICIP). Athens, Greece: IEEE, 2018: 3933-3937.
[19]	YAN Y F. Color image quality measures and retrieval[D]. New Jersey, USA: New Jersey Insititute of Techonlogy, 2006.
[20]	RUDERMAN D L, CRONIN T W, CHIAO C C. Statistics of cone responses to natural images: Implications for visual coding[J]. Journal of the Optical Society of America A, 1998, 15(8):2036. doi: 10.1364/JOSAA.15.002036 URL
[21]	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]// Computer Vision-ECCV 2016. Amsterdam, The Netherlands: ECCV, 2016: 694-711.
[22]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].(2014 -09-02) [2020-10-13]. https://www.researchgate.net/publication/319770291_Very_Deep_Convolutional_Networks_for_Large-Scale_Image_Recognition.
[23]	GHANI A S A, ISA N A M. Underwater image quality enhancement through composition of dual-intensity images and Rayleigh-stretching[C]// 2014 IEEE Fourth International Conference on Consumer Electronics Berlin (ICCE-Berlin). Berlin, Germany: IEEE, 2014: 219-220.
[24]	HUANG D M, WANG Y, SONG W, et al. Shallow-water image enhancement using relative global histogram stretching based on adaptive parameter acquisition[C]//International Conference on Multimedia Modeling. Bangkok, Thailand: MMM, 2018: 453-465.
[25]	ANCUTI C, ANCUTI C O, HABER T, et al. Enhancing underwater images and videos by fusion [C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA: IEEE, 2012: 81-88.
[26]	ANCUTI C O, ANCUTI C, DE VLEESCHOUWER C, et al. Color balance and fusion for underwater image enhancement[J]. IEEE Transactions on Image Processing, 2018, 27(1):379-393. doi: 10.1109/TIP.83 URL
[27]	LI C Y, ANWAR S, PORIKLI F. Underwater scene prior inspired deep underwater image and video enhancement[J]. Pattern Recognition, 2020, 98:107038. doi: 10.1016/j.patcog.2019.107038 URL
[28]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]//2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 2242-2251.

损失函数形式	L_GAN	L_GAN+U	L_GAN+C	L_all
PSNR	23.601	23.938	24.462	24.749
SSIM	0.711	0.732	0.745	0.764

算法	PSNR	SSIM	UIQM	CCF	信息熵
原图	17.180	0.628	2.589	24.285	7.135
文献[23]	14.782	0.567	2.892	23.308	7.526
文献[24]	18.063	0.686	2.409	26.919	7.547
文献[25]	16.507	0.657	2.699	25.787	7.744
文献[26]	18.612	0.714	3.216	26.866	7.519
文献[13]	18.776	0.698	3.219	27.710	7.262
文献[27]	16.153	0.588	2.995	16.997	6.749
本文算法	26.094	0.835	3.378	31.139	7.786

算法	场景	UIQM	CCF	信息熵
文献[23]	场景1	3.079	20.426	7.507
	场景2	2.420	19.494	7.051
	场景3	3.032	21.242	7.343
	场景4	3.382	21.134	7.454
	场景5	2.898	15.489	7.233
文献[24]	场景1	2.904	21.673	7.494
	场景2	2.011	21.904	7.168
	场景3	3.059	21.286	7.519
	场景4	3.293	20.427	7.329
	场景5	2.690	16.985	7.305
文献[25]	场景1	3.022	22.459	7.549
	场景2	2.623	21.427	7.247
	场景3	3.050	22.523	7.625
	场景4	3.340	21.842	7.472
	场景5	2.875	17.364	7.357
文献[26]	场景1	3.225	23.419	7.484
	场景2	2.505	20.444	7.232
	场景3	3.121	20.302	7.532

算法	场景	UIQM	CCF	信息熵
	场景4	3.429	23.140	7.467
	场景5	3.042	16.458	7.262
文献[13]	场景1	2.541	17.698	6.623
	场景2	2.851	19.053	6.997
	场景3	2.669	12.017	6.459
	场景4	3.198	14.953	6.581
	场景5	2.270	10.881	6.244
文献[27]	场景1	2.418	12.589	6.403
	场景2	3.108	16.396	6.846
	场景3	2.668	14.434	6.926
	场景4	2.912	11.469	6.162
	场景5	2.560	13.142	6.917
本文算法	场景1	3.421	25.313	7.701
	场景2	3.307	23.828	7.698
	场景3	3.272	24.259	7.698
	场景4	3.477	26.109	7.731
	场景5	3.194	19.491	7.444