Weld Defect Monitoring Based on Two-Stage Convolutional  Neural Network

doi:10.1007/s12204-023-2608-9

Abstract

Abstract: Zn vapour is easily generated on the surface by fusion welding galvanized steel sheet, resulting in the formation of defects. Rapidly developing computer vision sensing technology collects weld images in the welding process, then obtains laser fringe information through digital image processing, identifies welding defects, and finally realizes online control of weld defects. The performance of a convolutional neural network is related to its structure and the quality of the input image. The acquired original images are labeled with LabelMe, and repeated attempts are made to determine the appropriate filtering and edge detection image preprocessing methods. Two-stage convolutional neural networks with different structures are built on the Tensorflow deep learning framework, different thresholds of intersection over union are set, and deep learning methods are used to evaluate the collected original images and the preprocessed images separately. Compared with the test results, the comprehensive performance of the improved feature pyramid networks algorithm based on the basic network VGG16 is lower than that of the basic network Resnet101. Edge detection of the image will significantly improve the accuracy of the model. Adding blur will reduce the accuracy of the model slightly; however, the overall performance of the improved algorithm is still relatively good, which proves the stability of the algorithm. The self-developed software inspection system can be used for image preprocessing and defect recognition, which can be used to record the number and location of typical defects in continuous welds.

Key words: defects monitoring, image preprocessing, Resnet101, feature pyramid network

摘要： 镀锌钢板熔接时容易在表面产生锌蒸气，从而形成缺陷。快速发展的计算机视觉传感技术在焊接过程中采集焊接图像，然后通过数字图像处理获得激光边缘信息，识别焊接缺陷，最终实现焊接缺陷的在线控制。卷积神经网络的性能与它的结构和输入图像的质量有关。用LabelMe对获取的原始图像进行标记，并反复尝试确定适当的过滤和边缘检测图像预处理方法。在Tensorflow深度学习框架上构建了不同结构的两级卷积神经网络，设置了不同的交集大于联合的阈值，并使用深度学习方法分别对采集的原始图像和预处理后的图像进行评估。与测试结果相比，基于基本网络VGG16的改进的特征金字塔网络算法的综合性能低于基本网络Resnet101图像的边缘检测将显著提高模型的准确性。增加模糊度会略微降低模型的准确性；但是，该改进算法的整体性能比较好，证明了该算法的稳定性。自主研发的软件检测系统可用于图像预处理和缺陷识别，可用于记录连续焊缝中典型缺陷的数量和位置。

关键词: 缺陷监测，图像预处理，Resnet101，特征金字塔网络

CLC Number:

TP183
TG409

Xiao Wenbo, Xiong Jiakai, Yu Lesheng, He Yinshui, Ma Guohong. Weld Defect Monitoring Based on Two-Stage Convolutional Neural Network[J]. J Shanghai Jiaotong Univ Sci, 2025, 30(2): 291-299.

References

[1] ZHANG L, LIU S, HAN H, et al. Studies on the formation process and anti-corrosion performance of polypyrrole film deposited on the surface of Q235 steel by an electrochemical method [J]. Surface and Coatings Technology, 2018, 341: 95-102.
[2] SHREYAS P, PANDA B, KUMAR R. Mechanical properties and microstructure of 316L-galvanized steel weld [J]. Materials Today: Proceedings, 2020, 23: 600- 607.
[3] SHARMA A, LEE S J, CHOI D Y, et al. Effect of brazing current and speed on the bead characteristics, microstructure, and mechanical properties of the arc brazed galvanized steel sheets [J]. Journal of Materials Processing Technology, 2017, 249: 212-220.
[4] SATTARPANAH KARGANROUDI S, FEUJOFACK KEMDA V B, BARKA N. A novel method of identifying porosity during laser welding of galvanized steels using microhardness pattern matrix [J]. Manufacturing Letters, 2020, 25: 98-101.
[5] DELGADO M, GUERRERO M, GARZA R. Resistance spot welding of galvanized hsla steels [C]//9th International Conference on Zinc and Zinc Alloy Coated Steel Sheet & 2nd Asia-Pacific Galvanizing Conference. Beijing: Chinese Society for Metals, 2013: 233-236.
[6] MA G, WU C, YE J, et al. Effect of graphene on microstructure and mechanical properties of U-MIGwelded galvanized steel [J]. Journal of Materials Science: Materials in Electronics, 2020, 31(22): 20332- 20344.
[7] XU Y, FANG G, LV N, et al. Computer vision technology for seam tracking in robotic GTAW and GMAW [J]. Robotics and Computer-Integrated Manufacturing, 2015, 32: 25-36.
[8] MAHADEVAN R, JAGAN A, PAVITHRAN L, et al. Intelligent welding by using machine learning techniques [J]. Materials Today: Proceedings, 2021, 46: 7402-7410.
[9] DAVID E, RUMELHART, GEOFFREY E. Learning representations by back-propagating errors [J]. Nature, 1986, 323(6088): 533-536.
[10] SATHIYA P, PANNEERSELVAM K, ABDUL JALEEL M Y. Optimization of laser welding process parameters for super austenitic stainless steel using artificial neural networks and genetic algorithm [J]. Materials & Design (1980-2015 ), 2012, 36: 490-498.
[11] VALAVANIS I, KOSMOPOULOS D. Multiclass defect detection and classification in weld radiographic images using geometric and texture features [J]. Expert Systems With Applications, 2010, 37(12): 7606-7614.
[12] WU Y, GUO D, LIU H, et al. An end-to-end learning method for industrial defect detection [J]. Assembly Automation, 2020, 40(1): 31-39.
[13] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradientbased learning applied to document recognition [J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks [J]. Communications of the ACM, 2017, 60(6): 84-90.
[15] MIAO R, SHAN Z T, ZHOU Q Y, et al. Real-time defect identification of narrow overlap welds and application based on convolutional neural networks [J]. Journal of Manufacturing Systems, 2022, 62: 800-810.
[16] XIA C, PAN Z, FEI Z, et al. Vision based defects detection for Keyhole TIG welding using deep learning with visual explanation [J]. Journal of Manufacturing Processes, 2020, 56: 845-855.
[17] YANG L,WANG H, HUO B, et al. An automatic welding defect location algorithm based on deep learning [J]. NDT & E International, 2021, 120: 102435.
[18] MA G, YU L, YUAN H, et al. A vision-based method for lap weld defects monitoring of galvanized steel sheets using convolutional neural network [J]. Journal of Manufacturing Processes, 2021, 64: 130-139.
[19] CHEN H, CEN Z, WANG C, et al. Image restoration via improved Wiener filter applied to optical sparse aperture systems [J]. Optik, 2017, 147: 350-359.
[20] NNOLIMU A. Automated crack segmentation via saturation channel thresholding, area classification and fusion of modified level set segmentation with Canny edge detection [J]. Heliyon, 2020, 6(12): e05748.
[21] ZHENG Y, IWANA B K, UCHIDA S. Mining the displacement of max-pooling for text recognition [J]. Pattern Recognition, 2019, 93: 558-569.
[22] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation [C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 580-587.
[23] HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.
[24] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[25] CHEN K, ZENG Z, YANG J. A deep region-based pyramid neural network for automatic detection and multi-classification of various surface defects of aluminum alloys [J]. Journal of Building Engineering, 2021, 43: 102523.
[26] DAI W, LI D, TANG D, et al. Deep learning assisted vision inspection of resistance spot welds [J]. Journal of Manufacturing Processes, 2021, 62: 262-274.
[27] NANDINI G S, SIVA KUMAR A P, CHIDANANDA K. Dropout technique for image classification based on extreme learning machine [J]. Global Transitions Proceedings, 2021, 2(1): 111-116.

Weld Defect Monitoring Based on Two-Stage Convolutional Neural Network

基于两阶段卷积神经网络的焊缝缺陷监测

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 11

Recommended Articles

Metrics

Comments

[1]	DONG Zhaoxian, YU Shuo, SHEN Yanming. Multi-Scale Dynamic Hypergraph Convolutional Network for Traffic Flow Forecasting [J]. J Shanghai Jiaotong Univ Sci, 2025, 30(5): 880-888.
[2]	WU Yalei, LI Jinghua, KONG Dehui, LI Qianxing, YIN Baocai. 3D Hand Pose Estimation Using Semantic Dynamic Hypergraph Convolutional Networks [J]. J Shanghai Jiaotong Univ Sci, 2025, 30(5): 855-865.
[3]	Ma Jin, Ren Ze, Zhang Tongtong, Ding Ying, Lu Yilei, Peng Yinghong. Transformer-Based Contrastive Learning Method for Automated Sleep Stages Classification [J]. J Shanghai Jiaotong Univ Sci, 2025, 30(4): 720-732.
[4]	KE Jing¹(柯晶), ZHU Junchao² (朱俊超), YANG Xin¹(杨鑫), ZHANG Haolin³ (张浩林), SUN Yuxiang¹(孙宇翔), WANG Jiayi¹(王嘉怡), LU Yizhou⁴(鲁亦舟), SHEN Yiqing⁵(沈逸卿), LIU Sheng⁶(刘晟), JIANG Fusong⁷(蒋伏松), HUANG Qin⁸(黄琴). TshFNA-Examiner: A Nuclei Segmentation and Cancer Assessment Framework for Thyroid Cytology Image [J]. J Shanghai Jiaotong Univ Sci, 2024, 29(6): 945-957.
[5]	LI Mingai^{1, 2∗} (李明爱), WEI Lina¹ (魏丽娜). Motor Imagery Classification Based on Plain Convolutional Neural Network and Linear Interpolation [J]. J Shanghai Jiaotong Univ Sci, 2024, 29(6): 958-966.
[6]	GENG Zongsheng¹ (耿宗盛), ZHAO Dongdong^1,2 (赵东东), ZHOU Xingwen¹ (周兴文), YAN Lei¹ (闫磊), YAN Shi^1,2∗ (阎石). Leader-Following Consensus of Multi-Agent Systems via Fully Distributed Event-Based Control [J]. J Shanghai Jiaotong Univ Sci, 2024, 29(4): 640-645.
[7]	LIU Zengmin (刘增敏), WANG Shentao(王申涛), YAO Lixiu(姚莉秀), CAI Yunze(蔡云泽). Online Multi-Object Tracking Under Moving Unmanned Aerial Vehicle Platform Based on Object Detection and Feature Extraction Network [J]. J Shanghai Jiaotong Univ Sci, 2024, 29(3): 388-399.
[8]	ZHANG Yanjun(张彦军), WANG Biyun(王碧云)，CAI Yunze (蔡云泽). Multi-Channel Based on Attention Network for Infrared Small Target Detection [J]. J Shanghai Jiaotong Univ Sci, 2024, 29(3): 414-427.
[9]	WANG Yujuan1 (王玉娟)，LI Wengang2 (李文刚)，LIU .Jianyong3 (刘建勇),CHEN Guangxue4 (陈广学),WANG Jun1*(汪军). Color Prediction Model of Gray Hybrid Multifilament Fabric [J]. J Shanghai Jiaotong Univ Sci, 2023, 28(6): 802-808.
[10]	LIU Zhuoran (刘卓然), ZHAO Xu∗ (赵旭). Multilevel Disparity Reconstruction Network for Real-Time Stereo Matching [J]. J Shanghai Jiaotong Univ Sci, 2022, 27(5): 715-722.
[11]	SU Chong∗ (宿翀), LÜ Jing (吕晶), ZHANG Danyang (张丹阳), LI Hongguang∗ (李宏光). Affective Preferences Mining Approach with Applications in Process Control [J]. J Shanghai Jiaotong Univ Sci, 2022, 27(5): 737-746.