Defect Classification of Weld Metallographic Structure Based on Data Augmentation of Poisson Fusion

doi:10.16183/j.cnki.jsjtu.2022.202

Abstract

Abstract:

The classification of the defects in welding applications based on the metallographic structure images plays an important part in industrial welding quality inspections. In order to improve the classification performance of defects in the weld metallographic structure images with a small sample dataset available (the amount of samples being less than 30), a Poisson fusion method is used for data augmentation of the defect images and the ResNet18_PRO network is proposed. Both of the methods notably improve the defects classification performance. During data augmentation, the defect area is extracted from original defect samples via digital image processing, and the defect area is fused with normal samples by the Poisson fusion method to generate new defect samples, thus increasing the number of defect samples. The model in this paper is improved based on the ResNet18 network. The downsampling structure is improved to reduce the information loss in the original downsampling structure, and an improved space pyramid pooling structure is added at the end of the network to integrate multi-scale feature information. The classification performance before and after data augmentation is compared by different classification models, which verifies the significant effect of the data augmentation on the classification performance. Meanwhile, the ablation experiment of the ResNet18_PRO is conducted to verify the effectiveness of the improved network structure and the training strategy. It is found that the average classification accuracy of ResNet18_PRO reaches 98.83% and the average F₁-score reaches 98.76%, which greatly improves the classification accuracy of metallographic structure defects. Finally, the network is trained and tested with another industrial defect dataset and obtains good classification results. These results show that the proposed network has a good robustness and practical application value.

Key words: weld metallographic structure, data augmentation, Poisson fusion, defect classification

CLC Number:

BAI Xiongfei, GONG Shuicheng, LI Xuesong, XU Bo, YANG Xiaoli, WANG Mingyan. Defect Classification of Weld Metallographic Structure Based on Data Augmentation of Poisson Fusion[J]. Journal of Shanghai Jiao Tong University, 2023, 57(10): 1316-1328.

Figures/Tables 22

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Tab.1

Tab.2

Metrics and formula

度量指标	公式
准确度	M_acc= $N T P + N T N N T P + N T N + N F P + N F N$
精确率	M_pre= $N T P N T P + N F P$
召回率	M_re= $N T P N T P + N F N$
F1-score	F₁= $2 M p r e M r e M p r e + M r e$

Tab.2

Fig.8

Fig.9

Tab.3

Testing result of all models on test dataset (original data)%

模型	准确度	$F 1 h o$	$F 1 c r$	$F 1 n o$	$bar{F}_{1}^{ma}$
VGG13+ELR	34.45	2.25	0.00	50.30	17.71
GoogLeNet+ELR	59.88	68.37	0.00	74.33	47.57
ResNet18+ELR	35.08	7.49	0.00	50.60	19.36
ResNet18_Pro+ELR	64.30	66.81	2.11	86.62	51.85

Tab.3

Fig.10

Fig.11

Tab.4

Testing result of all models on test dataset (after data augmentation)%

模型	准确度	$F 1 h o$	$F 1 c r$	$F 1 n o$	$\bar{F}_{1}^{ma}$
VGG13+ELR	95.78	95.61	92.63	98.33	95.52
GoogLeNet+ELR	97.85	97.74	96.56	99.07	97.86
ResNet18+ELR	96.56	96.91	94.57	97.61	96.36
ResNet18_PRO+ELR	98.83	98.88	97.96	99.45	98.76

Tab.4

Fig.12

Tab.5

Tab.6

Tab.7

Result of ablation experiment%

类型	模型	准确度	$F 1 h o$	$F 1 c r$	$F 1 n o$	$\bar{F}_{1}^{ma}$
Baseline	ResNet18	85.31	77.39	80.92	95.30	84.54
结构改进	+ ISPP	93.67(+8.36)	93.01(+15.62)	90.02(+9.10)	97.68(+2.38)	93.57(+9.03)
	+LDPS	96.37(+11.06)	95.97(+18.58)	94.72(+13.80)	97.49(+2.19)	96.06(+11.52)
	+ ISPP +LDPS (ResNet18_PRO)	97.34(+12.03)	96.19(+18.83)	96.24(+15.32)	98.57(+3.27)	97.00(+12.46)
策略改进	ResNet18 +ELR	96.56(+11.25)	96.91(+19.52)	94.57(+13.65)	97.61(+2.31)	96.36(+11.82)
	ResNet18_PRO+ELR	98.83(+13.52)	98.88(+21.49)	97.96(+17.04)	99.45(+4.15)	98.76(+14.22)

Tab.7

Fig.13

Fig.14

Tab.8

References 25

[1]	周正干, 滕升华, 江巍, 等. 焊缝X射线检测及其结果的评判方法综述[J]. 焊接学报, 2002, 23(3): 85-88.
	ZHOU Zhenggan, TENG Shenghua, JIANG Wei, et al. Research on defect detection and evaluation in WeldS with X-rays[J]. Transactions of the China Welding Institution, 2002, 23(3): 85-88.
[2]	邵中华. X射线焊缝数字图像的缺陷提取技术研究[D]. 太原: 中北大学, 2011.
	SHAO Zhonghua. Research on extraction technology of weld defects in X-ray image[D]. Taiyuan: North University of China, 2011.
[3]	王思宇, 高炜欣, 张翔松. X射线焊缝图像缺陷检测算法综述[J]. 热加工工艺, 2020, 49(15): 1-8.
	WANG Siyu, GAO Weixin, ZHANG Xiangsong. Summary of defect detection algorithms for X-ray weld image[J]. Hot Working Technology, 2020, 49(15): 1-8.
[4]	胡丹, 高向东, 张南峰, 等. 焊缝缺陷检测现状与展望综述[J]. 机电工程, 2020, 37(7): 736-742.
	HU Dan, GAO Xiangdong, ZHANG Nanfeng, et al. Review of status and prospect of weld defect detection[J]. Journal of Mechanical & Electrical Engineering, 2020, 37(7): 736-742.
[5]	刘玉宗. 焊丝种类对6082-T6铝合金激光-电弧复合焊接焊缝质量的影响研究[J]. 光源与照明, 2021(2): 38-39.
	LIU Yuzong. Study on the influence of weld wire types on the quality of 6082-T6 aluminum alloy laser-arc composite weld welds[J]. Lamps & Lighting, 2021(2): 38-39.
[6]	DUAN F, YIN S F, SONG P P, et al. Automatic welding defect detection of X-ray images by using cascade AdaBoost with penalty term[J]. IEEE Access, 7: 125929-125938. doi: 10.1109/Access.6287639 URL
[7]	HOU W H, WEI Y, GUO J, et al. Automatic detection of welding defects using deep neural network[J]. Journal of Physics: Conference Series, 2018, 933: 012006. doi: 10.1088/1742-6596/933/1/012006 URL
[8]	AJMI C, ZAPATA J, MARTÍNEZ-ÁLVAREZ J J, et al. Using deep learning for defect classification on a small weld X-ray image dataset[J]. Journal of Nondestructive Evaluation, 2020, 39(3): 1-13.
[9]	闫耀东. 基于深度学习的不锈钢焊缝缺陷分类研究[D]. 太原: 太原科技大学, 2021.
	YAN Yaodong. Research on defect classification of stainless steel welds based on deep learning[D]. Taiyuan: Taiyuan University of Science and Technology, 2021.
[10]	罗爱民, 沈才洪, 易彬, 等. 基于改进二叉树多分类SVM的焊缝缺陷分类方法[J]. 焊接学报, 2010, 31(7): 51-54.
	LUO Aimin, SHEN Caihong, YI Bin, et al. Method of multi-classification by improved binary tree based on SVM for welding defects recongnition[J]. Transactions of the China Welding Institution, 2010, 31(7): 51-54.
[11]	刘欢, 刘骁佳, 王宇斐, 等. 基于复合卷积层神经网络结构的焊缝缺陷分类技术[J]. 航空学报, 2022, 43(Sup.1): 165-172.
	LIU Huan, LIU Xiaojia, WANG Yufei, et al. Weld defect classification technology based on compound convolution neural network structure[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43 (Sup.1): 165-172.
[12]	谷静, 张可帅, 朱漪曼. 基于卷积神经网络的焊缝缺陷图像分类研究[J]. 应用光学, 2020, 41(3): 531-537. doi: 10.5768/JAO202041.0302007
	GU Jing, ZHANG Keshuai, ZHU Yiman. Research on weld defect image classification based on convolutional neural network[J]. Journal of Applied Optics, 2020, 41(3): 531-537. doi: 10.5768/JAO202041.0302007
[13]	葛轶洲, 刘恒, 王言, 等. 小样本困境下的深度学习图像识别综述[J]. 软件学报, 2022, 33(1): 193-210.
	GE Yizhou, LIU Heng, WANG Yan, et al. Survey on deep learning image recognition in dilemma of small samples[J]. Journal of Software, 2022, 33(1): 193-210.
[14]	李钧正, 殷子玉, 乐心怡. 基于小样本学习的钢板表面缺陷检测技术[J]. 航空科学技术, 2021, 32(6): 65-70.
	LI Junzheng, YIN Ziyu, LE Xinyi. Surface defect detection for steel plate with small dataset[J]. Aeronautical Science & Technology, 2021, 32(6): 65-70.
[15]	CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: Synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16: 321-357. doi: 10.1613/jair.953 URL
[16]	宋宗垚. 小样本数据下的工业无损检测图像损伤定位算法的研究[D]. 天津: 天津大学, 2019.
	SONG Zongyao. Damage location algorithm for few-shot industrial nondestructive testing[D]. Tianjin: Tianjin University, 2019.
[17]	马岭, 鲁越, 蒋慧琴, 等. 基于小样本学习的LCD产品缺陷自动检测方法[J]. 智能系统学报, 2020, 15(3): 560-567.
	MA Ling, LU Yue, JIANG Huiqin, et al. An automatic small sample learning-based detection method for LCD product defects[J]. CAAI Transactions on Intelligent Systems, 2020, 15(3): 560-567.
[18]	PÉREZ P, GANGNET M, BLAKE A. Poisson image editing[J]. ACM Transactions on Graphics, 2003, 22(3): 313-318. doi: 10.1145/882262.882269 URL
[19]	左飞. 图像处理中的数学修炼[M]. 第2版. 北京: 清华大学出版社, 2020.
	ZUO Fei. Applied mathematics in digital image processing[M]. 2nd ed. Beijing: Tsinghua University Press, 2020.
[20]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[DB/OL]. (2015-12-10)[2022-08-03].https://arxiv.org/abs/1512.03385.
[21]	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. doi: 10.1109/TPAMI.2015.2389824 pmid: 26353135
[22]	LIN M, CHEN Q, YAN S. Network in network[DB/OL]. (2014-03-04)[2022-08-03]. https://arxiv.org/abs/1312.4400.
[23]	国家标准局.金属熔化焊焊缝缺陷分类及说明: GB/T 6417—1986[S]. 北京: 国家标准局, 1986.
	National Bureau of Standards.Classification of imperfections in metallic fusion welds, with explanations: GB/T 6417—1986[S]. Beijing: National Bureau of Standards, 1986.
[24]	BRANCO P, TORGO L, RIBEIRO R P. A survey of predictive modeling on imbalanced domains[J]. ACM Computing Surveys, 2016, 49(2): 1-50.
[25]	SUN Y M, WONG A K C, KAMEL M S. Classification of imbalanced data: A review[J]. International Journal of Pattern Recognition and Artificial Intelligence, 2009, 23(4): 687-719. doi: 10.1142/S0218001409007326 URL

图片类型	原始样本数	增强后的样本数
气孔	81	950
裂纹	26	946
正常	966	966

来源	真实标签	VGG13+ELR	GoogLeNet+ELR	ResNet18+ELR	ResNet18_PRO+ELR
图12(a)	气孔	裂纹	裂纹	裂纹	气孔
图12(b)	裂纹	正常	正常	正常	裂纹

分类方法	准确度/%	分类速度/(帧·s^-1)
人工分类	99	约1
ResNet18_PRO	98.83	GPU:约100 CPU:约10

模型	分类准确度/%	$\bar{F}_{1}^{ma}$/%
VGG13+ELR	94.90	97.43
GoogLeNet+ELR	97.48	98.62
ResNet18+ELR	98.53	98.27
ResNet18_PRO+ELR	98.96	99.21