基于泊松融合数据增强的焊缝金相组织缺陷分类研究

doi:10.16183/j.cnki.jsjtu.2022.202

上海交通大学学报 ›› 2023, Vol. 57 ›› Issue (10): 1316-1328.doi: 10.16183/j.cnki.jsjtu.2022.202

所属专题：《上海交通大学学报》2023年“机械与动力工程”专题

基于泊松融合数据增强的焊缝金相组织缺陷分类研究

白雄飞¹, 龚水成¹, 李雪松¹^,³(), 许博², 杨晓力³, 王明彦³

1.上海交通大学机械与动力工程学院, 上海 200240
2.上汽大众汽车有限公司, 上海 201805
3.湖南华研实验室有限公司, 湖南湘潭 411000

收稿日期:2022-05-31 修回日期:2022-08-04 接受日期:2022-08-26 出版日期:2023-10-28 发布日期:2023-10-31
通讯作者: 李雪松 E-mail:xuesonl@sjtu.edu.cn
作者简介:白雄飞(1999-),硕士生,从事目标检测与目标分割研究.
基金资助:
国家自然科学基金(E52006140);湖南省科技创新计划资助项目(2020RC5021)

Defect Classification of Weld Metallographic Structure Based on Data Augmentation of Poisson Fusion

BAI Xiongfei¹, GONG Shuicheng¹, LI Xuesong¹^,³(), XU Bo², YANG Xiaoli³, WANG Mingyan³

1. School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
2. SAIC Volkswagen Automotive Co., Ltd., Shanghai 201805, China
3. Hunan Huayan Laboratory Co., Ltd., Xiangtan 411000, Hunan, China

Received:2022-05-31 Revised:2022-08-04 Accepted:2022-08-26 Online:2023-10-28 Published:2023-10-31
Contact: LI Xuesong E-mail:xuesonl@sjtu.edu.cn

摘要/Abstract

摘要：

基于焊缝金相组织图像对焊缝内部缺陷进行分类,是工业焊接质量检测的重要一环.为提高小样本(样本数不大于30)焊缝金相组织图像中缺陷的分类效果,采用泊松融合方法对缺陷图像进行数据增强,提出ResNet18_PRO分类网络模型,显著提升缺陷分类精度.数据增强方面,通过数字图像处理的方法提取出原缺陷样本中的缺陷区域,后利用泊松融合方法将缺陷区域与正常样本进行融合从而生成新的缺陷样本,以此扩充缺陷样本数据;网络模型方面,在ResNet18网络模型基础上,对其下采样结构进行改进,以减少原下采样结构带来的信息损失,同时在网络末端增加改进的空间金字塔池化(ISPP)结构,以整合多尺度的特征信息.通过多个分类模型对样本扩充前后的缺陷分类效果进行对比,验证了该数据增强方法对分类效果的提升具有较为显著的作用,同时对ResNet18_PRO网络模型进行消融实验,验证了网络各改进部分及训练策略改进的有效性.ResNet18_PRO模型对增强后的数据平均分类准确度达到98.83%,平均F₁分数达到98.76%,显著提高了金相组织缺陷的分类效果,将该模型运用于其他工业缺陷数据集取得了较好的分类效果.实验结果表明,该模型有良好的鲁棒性,具有较好的实用价值.

关键词: 焊缝金相组织, 数据增强, 泊松融合, 缺陷分类

Abstract:

The classification of the defects in welding applications based on the metallographic structure images plays an important part in industrial welding quality inspections. In order to improve the classification performance of defects in the weld metallographic structure images with a small sample dataset available (the amount of samples being less than 30), a Poisson fusion method is used for data augmentation of the defect images and the ResNet18_PRO network is proposed. Both of the methods notably improve the defects classification performance. During data augmentation, the defect area is extracted from original defect samples via digital image processing, and the defect area is fused with normal samples by the Poisson fusion method to generate new defect samples, thus increasing the number of defect samples. The model in this paper is improved based on the ResNet18 network. The downsampling structure is improved to reduce the information loss in the original downsampling structure, and an improved space pyramid pooling structure is added at the end of the network to integrate multi-scale feature information. The classification performance before and after data augmentation is compared by different classification models, which verifies the significant effect of the data augmentation on the classification performance. Meanwhile, the ablation experiment of the ResNet18_PRO is conducted to verify the effectiveness of the improved network structure and the training strategy. It is found that the average classification accuracy of ResNet18_PRO reaches 98.83% and the average F₁-score reaches 98.76%, which greatly improves the classification accuracy of metallographic structure defects. Finally, the network is trained and tested with another industrial defect dataset and obtains good classification results. These results show that the proposed network has a good robustness and practical application value.

Key words: weld metallographic structure, data augmentation, Poisson fusion, defect classification

中图分类号:

白雄飞, 龚水成, 李雪松, 许博, 杨晓力, 王明彦. 基于泊松融合数据增强的焊缝金相组织缺陷分类研究[J]. 上海交通大学学报, 2023, 57(10): 1316-1328.

BAI Xiongfei, GONG Shuicheng, LI Xuesong, XU Bo, YANG Xiaoli, WANG Mingyan. Defect Classification of Weld Metallographic Structure Based on Data Augmentation of Poisson Fusion[J]. Journal of Shanghai Jiao Tong University, 2023, 57(10): 1316-1328.

图/表 22

图1

图2

图3

图4

图5

图6

图7

表1

表2

评价指标及公式

度量指标	公式
准确度	M_acc= $N T P + N T N N T P + N T N + N F P + N F N$
精确率	M_pre= $N T P N T P + N F P$
召回率	M_re= $N T P N T P + N F N$
F1-score	F₁= $2 M p r e M r e M p r e + M r e$

表2

图8

图9

表3

各模型在测试集上的结果(原始数据)

模型	准确度	$F 1 h o$	$F 1 c r$	$F 1 n o$	$bar{F}_{1}^{ma}$
VGG13+ELR	34.45	2.25	0.00	50.30	17.71
GoogLeNet+ELR	59.88	68.37	0.00	74.33	47.57
ResNet18+ELR	35.08	7.49	0.00	50.60	19.36
ResNet18_Pro+ELR	64.30	66.81	2.11	86.62	51.85

表3

图10

图11

表4

各模型在测试集上的结果(数据增强后)

模型	准确度	$F 1 h o$	$F 1 c r$	$F 1 n o$	$\bar{F}_{1}^{ma}$
VGG13+ELR	95.78	95.61	92.63	98.33	95.52
GoogLeNet+ELR	97.85	97.74	96.56	99.07	97.86
ResNet18+ELR	96.56	96.91	94.57	97.61	96.36
ResNet18_PRO+ELR	98.83	98.88	97.96	99.45	98.76

表4

图12

表5

表6

表7

消融实验结果

类型	模型	准确度	$F 1 h o$	$F 1 c r$	$F 1 n o$	$\bar{F}_{1}^{ma}$
Baseline	ResNet18	85.31	77.39	80.92	95.30	84.54
结构改进	+ ISPP	93.67(+8.36)	93.01(+15.62)	90.02(+9.10)	97.68(+2.38)	93.57(+9.03)
	+LDPS	96.37(+11.06)	95.97(+18.58)	94.72(+13.80)	97.49(+2.19)	96.06(+11.52)
	+ ISPP +LDPS (ResNet18_PRO)	97.34(+12.03)	96.19(+18.83)	96.24(+15.32)	98.57(+3.27)	97.00(+12.46)
策略改进	ResNet18 +ELR	96.56(+11.25)	96.91(+19.52)	94.57(+13.65)	97.61(+2.31)	96.36(+11.82)
	ResNet18_PRO+ELR	98.83(+13.52)	98.88(+21.49)	97.96(+17.04)	99.45(+4.15)	98.76(+14.22)

表7

图13

图14

表8

参考文献 25

[1]	周正干, 滕升华, 江巍, 等. 焊缝X射线检测及其结果的评判方法综述[J]. 焊接学报, 2002, 23(3): 85-88.
	ZHOU Zhenggan, TENG Shenghua, JIANG Wei, et al. Research on defect detection and evaluation in WeldS with X-rays[J]. Transactions of the China Welding Institution, 2002, 23(3): 85-88.
[2]	邵中华. X射线焊缝数字图像的缺陷提取技术研究[D]. 太原: 中北大学, 2011.
	SHAO Zhonghua. Research on extraction technology of weld defects in X-ray image[D]. Taiyuan: North University of China, 2011.
[3]	王思宇, 高炜欣, 张翔松. X射线焊缝图像缺陷检测算法综述[J]. 热加工工艺, 2020, 49(15): 1-8.
	WANG Siyu, GAO Weixin, ZHANG Xiangsong. Summary of defect detection algorithms for X-ray weld image[J]. Hot Working Technology, 2020, 49(15): 1-8.
[4]	胡丹, 高向东, 张南峰, 等. 焊缝缺陷检测现状与展望综述[J]. 机电工程, 2020, 37(7): 736-742.
	HU Dan, GAO Xiangdong, ZHANG Nanfeng, et al. Review of status and prospect of weld defect detection[J]. Journal of Mechanical & Electrical Engineering, 2020, 37(7): 736-742.
[5]	刘玉宗. 焊丝种类对6082-T6铝合金激光-电弧复合焊接焊缝质量的影响研究[J]. 光源与照明, 2021(2): 38-39.
	LIU Yuzong. Study on the influence of weld wire types on the quality of 6082-T6 aluminum alloy laser-arc composite weld welds[J]. Lamps & Lighting, 2021(2): 38-39.
[6]	DUAN F, YIN S F, SONG P P, et al. Automatic welding defect detection of X-ray images by using cascade AdaBoost with penalty term[J]. IEEE Access, 7: 125929-125938. doi: 10.1109/Access.6287639 URL
[7]	HOU W H, WEI Y, GUO J, et al. Automatic detection of welding defects using deep neural network[J]. Journal of Physics: Conference Series, 2018, 933: 012006. doi: 10.1088/1742-6596/933/1/012006 URL
[8]	AJMI C, ZAPATA J, MARTÍNEZ-ÁLVAREZ J J, et al. Using deep learning for defect classification on a small weld X-ray image dataset[J]. Journal of Nondestructive Evaluation, 2020, 39(3): 1-13.
[9]	闫耀东. 基于深度学习的不锈钢焊缝缺陷分类研究[D]. 太原: 太原科技大学, 2021.
	YAN Yaodong. Research on defect classification of stainless steel welds based on deep learning[D]. Taiyuan: Taiyuan University of Science and Technology, 2021.
[10]	罗爱民, 沈才洪, 易彬, 等. 基于改进二叉树多分类SVM的焊缝缺陷分类方法[J]. 焊接学报, 2010, 31(7): 51-54.
	LUO Aimin, SHEN Caihong, YI Bin, et al. Method of multi-classification by improved binary tree based on SVM for welding defects recongnition[J]. Transactions of the China Welding Institution, 2010, 31(7): 51-54.
[11]	刘欢, 刘骁佳, 王宇斐, 等. 基于复合卷积层神经网络结构的焊缝缺陷分类技术[J]. 航空学报, 2022, 43(Sup.1): 165-172.
	LIU Huan, LIU Xiaojia, WANG Yufei, et al. Weld defect classification technology based on compound convolution neural network structure[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43 (Sup.1): 165-172.
[12]	谷静, 张可帅, 朱漪曼. 基于卷积神经网络的焊缝缺陷图像分类研究[J]. 应用光学, 2020, 41(3): 531-537. doi: 10.5768/JAO202041.0302007
	GU Jing, ZHANG Keshuai, ZHU Yiman. Research on weld defect image classification based on convolutional neural network[J]. Journal of Applied Optics, 2020, 41(3): 531-537. doi: 10.5768/JAO202041.0302007
[13]	葛轶洲, 刘恒, 王言, 等. 小样本困境下的深度学习图像识别综述[J]. 软件学报, 2022, 33(1): 193-210.
	GE Yizhou, LIU Heng, WANG Yan, et al. Survey on deep learning image recognition in dilemma of small samples[J]. Journal of Software, 2022, 33(1): 193-210.
[14]	李钧正, 殷子玉, 乐心怡. 基于小样本学习的钢板表面缺陷检测技术[J]. 航空科学技术, 2021, 32(6): 65-70.
	LI Junzheng, YIN Ziyu, LE Xinyi. Surface defect detection for steel plate with small dataset[J]. Aeronautical Science & Technology, 2021, 32(6): 65-70.
[15]	CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: Synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16: 321-357. doi: 10.1613/jair.953 URL
[16]	宋宗垚. 小样本数据下的工业无损检测图像损伤定位算法的研究[D]. 天津: 天津大学, 2019.
	SONG Zongyao. Damage location algorithm for few-shot industrial nondestructive testing[D]. Tianjin: Tianjin University, 2019.
[17]	马岭, 鲁越, 蒋慧琴, 等. 基于小样本学习的LCD产品缺陷自动检测方法[J]. 智能系统学报, 2020, 15(3): 560-567.
	MA Ling, LU Yue, JIANG Huiqin, et al. An automatic small sample learning-based detection method for LCD product defects[J]. CAAI Transactions on Intelligent Systems, 2020, 15(3): 560-567.
[18]	PÉREZ P, GANGNET M, BLAKE A. Poisson image editing[J]. ACM Transactions on Graphics, 2003, 22(3): 313-318. doi: 10.1145/882262.882269 URL
[19]	左飞. 图像处理中的数学修炼[M]. 第2版. 北京: 清华大学出版社, 2020.
	ZUO Fei. Applied mathematics in digital image processing[M]. 2nd ed. Beijing: Tsinghua University Press, 2020.
[20]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[DB/OL]. (2015-12-10)[2022-08-03].https://arxiv.org/abs/1512.03385.
[21]	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. doi: 10.1109/TPAMI.2015.2389824 pmid: 26353135
[22]	LIN M, CHEN Q, YAN S. Network in network[DB/OL]. (2014-03-04)[2022-08-03]. https://arxiv.org/abs/1312.4400.
[23]	国家标准局.金属熔化焊焊缝缺陷分类及说明: GB/T 6417—1986[S]. 北京: 国家标准局, 1986.
	National Bureau of Standards.Classification of imperfections in metallic fusion welds, with explanations: GB/T 6417—1986[S]. Beijing: National Bureau of Standards, 1986.
[24]	BRANCO P, TORGO L, RIBEIRO R P. A survey of predictive modeling on imbalanced domains[J]. ACM Computing Surveys, 2016, 49(2): 1-50.
[25]	SUN Y M, WONG A K C, KAMEL M S. Classification of imbalanced data: A review[J]. International Journal of Pattern Recognition and Artificial Intelligence, 2009, 23(4): 687-719. doi: 10.1142/S0218001409007326 URL

图片类型	原始样本数	增强后的样本数
气孔	81	950
裂纹	26	946
正常	966	966

来源	真实标签	VGG13+ELR	GoogLeNet+ELR	ResNet18+ELR	ResNet18_PRO+ELR
图12(a)	气孔	裂纹	裂纹	裂纹	气孔
图12(b)	裂纹	正常	正常	正常	裂纹

分类方法	准确度/%	分类速度/(帧·s^-1)
人工分类	99	约1
ResNet18_PRO	98.83	GPU:约100 CPU:约10

模型	分类准确度/%	$\bar{F}_{1}^{ma}$/%
VGG13+ELR	94.90	97.43
GoogLeNet+ELR	97.48	98.62
ResNet18+ELR	98.53	98.27
ResNet18_PRO+ELR	98.96	99.21

基于泊松融合数据增强的焊缝金相组织缺陷分类研究

Defect Classification of Weld Metallographic Structure Based on Data Augmentation of Poisson Fusion

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 22

参考文献 25

相关文章 1

编辑推荐

Metrics

本文评价