Multiple Detection Model Fusion Framework for Printed Circuit Board Defect Detection

doi:10.1007/s12204-022-2471-0

Abstract

Abstract: The printed circuit board (PCB) is an indispensable component of electronic products, which determines the quality of these products. With the development and advancement of manufacturing technology, the layout and structure of PCB are getting complicated. However, there are few effective and accurate PCB defect detection methods. There are high requirements for the accuracy of PCB defect detection in the actual production environment, so we propose two PCB defect detection frameworks with multiple model fusion including the defect detection by multi-model voting method (DDMV) and the defect detection by multi-model learning method (DDML). With the purpose of reducing wrong and missing detection, the DDMV and DDML integrate multiple defect detection networks with different fusion strategies. The effectiveness and accuracy of the proposed framework are verified with extensive experiments on two open-source PCB datasets. The experimental results demonstrate that the proposed DDMV and DDML are better than any other individual state-of-the-art PCB defect detection model in F1-score, and the area under curve value of DDML is also higher than that of any other individual detection model. Furthermore, compared with DDMV, the DDML with an automatic machine learning method achieves the best performance in PCB defect detection, and the F1-score on the two datasets can reach 99.7% and 95.6% respectively.

Key words: printed circuit board (PCB), defect detection, model fusion, object detection model

摘要： 印刷电路板（PCB）是电子产品不可或缺的组成部分，它决定了这些产品的质量。随着制造技术的发展和进步，PCB的布局和结构变得越来越复杂。然而，有效准确的PCB缺陷检测方法却很少。在实际生产环境中，对PCB缺陷检测的准确性有很高的要求，因此我们提出了两种包括多模型融合的PCB缺陷检测框架，包括多模型投票方法（DDMV）和多模型学习方法（DDML）的缺陷检测。为了减少错误和漏检，DDMV和DDML将多个具有不同融合策略的缺陷检测网络进行整合。通过对两个开源PCB数据集进行大量实验，验证了所提出框架的有效性和准确性。实验结果表明，所提出的DDMV和DDML在F1分数方面优于其他任何单独的最先进的PCB缺陷检测模型，而DDML的曲线下面积值也高于其他任何单独的检测模型。此外，与DDMV相比，使用自动机器学习方法的DDML在PCB缺陷检测方面达到了最好的性能，在两个数据集上的F1分数分别可达到99.7%和95.6%。

关键词: 印刷电路板（PCB），缺陷检测，模型融合，目标检测模型

CLC Number:

TP399

WU Xingl(武星), ZHANG Qingfeng(张庆丰), WANG Jianjia(王健嘉), YAO Junfeng(姚骏峰), Guo Yike.(郭毅可). Multiple Detection Model Fusion Framework for Printed Circuit Board Defect Detection[J]. J Shanghai Jiaotong Univ Sci, 2023, 28(6): 717-727.

References 37

[8]	LIN T Y, DOLL′ AR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 936-944.
[1]	CHAUDHARY V, DAVE I R, UPLA K P. Automatic visual inspection of printed circuit board for defect detection and classification [C]//2017 International Conference on Wireless Communications, Signal Processing and Networking. Chennai: IEEE, 2017: 732-737.
[9]	JIANG H Z, LEARNED-MILLER E. Face detection with the faster R-CNN [C]//2017 12th IEEE International Conference on Automatic Face & Gesture Recognition. Washington, DC, USA: IEEE, 2017: 650-657.
[2]	ZHU J H, WU A, LIU X P. Printed circuit board defect visual detection based on wavelet denoising [J]. IOP Conference Series: Materials Science and Engineering, 2018, 392: 062055.
[10]	WANG Y, LUO X B, DING L, et al. Detection based visual tracking with convolutional neural network [J]. Knowledge-Based Systems, 2019, 175: 62-71.
[3]	KUO C F J, FANG T Y, LEE C L, et al. Automated optical inspection system for surface mount device light emitting diodes [J]. Journal of Intelligent Manufacturing, 2019, 30(2): 641-655.
[11]	WEI H, YANG C Z, YU Q. Efficient graph-based search for object detection [J]. Information Sciences, 2017, 385/386: 395-414.
[4]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks [J]. Communications of the ACM, 2017, 60(6): 84-90.
[12]	BRIA A, MARROCCO C, MOLINARA M, et al. An effective learning strategy for cascaded object detection [J]. Information Sciences, 2016, 340/341: 17-26.
[5]	WU X, ZHONG M Y, GUO Y K, et al. The assessment of small bowel motility with attentive deformable neural network [J]. Information Sciences, 2020, 508: 22-32.
[13]	OLSON R S, MOORE J H. TPOT: A tree-based pipeline optimization tool for automating machine learning [M]//Automated machine learning. Cham: Springer, 2019: 151-160.
[6]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single shot multibox detector [M]//Computer vision -ECCV 2016. Cham: Springer, 2016: 21-37.
[14]	HOSANG J, BENENSON R, SCHIELE B. Learning non-maximum suppression [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 6469-6477.
[7]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: Unified, real-time object detection [C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE, 2016: 779- 788.
[15]	DING R W, DAI L H, LI G P, et al. TDD-net: a tiny defect detection network for printed circuit boards [J].CAAI Transactions on Intelligence Technology, 2019, 4(2): 110-116.
[8]	LIN T Y, DOLL′ AR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 936-944.
[16]	TANG S L, HE F, HUANG X L, et al. Online PCB defect detector on a new PCB defect dataset [DB/OL]. (2019-02-17). https://arxiv.org/abs/1902.06197.
[9]	JIANG H Z, LEARNED-MILLER E. Face detection with the faster R-CNN [C]//2017 12th IEEE International Conference on Automatic Face & Gesture Recognition. Washington, DC, USA: IEEE, 2017: 650-657.
[17]	DALAL N, TRIGGS B. Histograms of oriented gradients for human detection [C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, CA: IEEE, 2005: 886-893.
[10]	WANG Y, LUO X B, DING L, et al. Detection based visual tracking with convolutional neural network [J]. Knowledge-Based Systems, 2019, 175: 62-71.
[18]	LOWE D G. Distinctive image features from scaleinvariant keypoints [J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[11]	WEI H, YANG C Z, YU Q. Efficient graph-based search for object detection [J]. Information Sciences, 2017, 385/386: 395-414.
[19]	BAY H, TUYTELAARS T, VAN GOOL L. SURF: speeded up robust features [M]//Computer vision - ECCV 2006. Berlin, Heidelberg: Springer, 2006: 404- 417.
[12]	BRIA A, MARROCCO C, MOLINARA M, et al. An effective learning strategy for cascaded object detection [J]. Information Sciences, 2016, 340/341: 17-26.
[20]	LU Z S, HE Q Q, XIANG X G, et al. Defect detection of PCB based on Bayes feature fusion [J]. The Journal of Engineering, 2018, 2018(16): 1741-1745.
[13]	OLSON R S, MOORE J H. TPOT: A tree-based pipeline optimization tool for automating machine learning [M]//Automated machine learning. Cham: Springer, 2019: 151-160.
[21]	BENEDEK C. Detection of soldering defects in printed circuit boards with hierarchical marked point processes [J]. Pattern Recognition Letters, 2011, 32(13): 1535- 1543.
[14]	HOSANG J, BENENSON R, SCHIELE B. Learning non-maximum suppression [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 6469-6477.
[22]	GAIDHANE V H, HOTE Y V, SINGH V. An efficient similarity measure approach for PCB surface defect detection [J]. Pattern Analysis and Applications, 2018, 21(1): 277-289.
[15]	DING R W, DAI L H, LI G P, et al. TDD-net: a tiny defect detection network for printed circuit boards [J].CAAI Transactions on Intelligence Technology, 2019, 4(2): 110-116.
[23]	ZHANG C, SHI W, LI X F, et al. Improved bare PCB defect detection approach based on deep feature learning [J]. The Journal of Engineering, 2018, 2018(16): 1415-1420.
[16]	TANG S L, HE F, HUANG X L, et al. Online PCB defect detector on a new PCB defect dataset [DB/OL]. (2019-02-17). https://arxiv.org/abs/1902.06197.
[24]	DAI W T, MUJEEB A, ERDT M, et al. Soldering defect detection in automatic optical inspection [J]. Advanced Engineering Informatics, 2020, 43: 101004.
[17]	DALAL N, TRIGGS B. Histograms of oriented gradients for human detection [C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, CA: IEEE, 2005: 886-893.
[25]	BENJDIRA B, KHURSHEED T, KOUBAA A, et al. Car detection using unmanned aerial vehicles: Comparison between faster R-CNN and YOLOv3 [C]//2019 1st International Conference on Unmanned Vehicle Systems-Oman. Muscat: IEEE, 2019: 1-6.
[18]	LOWE D G. Distinctive image features from scaleinvariant keypoints [J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[26]	LEI H W, WANG B, WU H H, et al. Defect detection for polymeric polarizer based on faster R-CNN [J]. Journal of Information Hiding and Multimedia Signal Processing, 2018, 9(6): 1414-1420.
[19]	BAY H, TUYTELAARS T, VAN GOOL L. SURF: speeded up robust features [M]//Computer vision - ECCV 2006. Berlin, Heidelberg: Springer, 2006: 404- 417.
[27]	LI Y T, HUANG H S, XIE Q S, et al. Research on a surface defect detection algorithm based on MobileNet-SSD [J]. Applied Sciences, 2018, 8(9): 1678.
[20]	LU Z S, HE Q Q, XIANG X G, et al. Defect detection of PCB based on Bayes feature fusion [J]. The Journal of Engineering, 2018, 2018(16): 1741-1745.
[28]	LI J Y, SU Z F, GENG J H, et al. Real-time detection of steel strip surface defects based on improved YOLO detection network [J]. IFAC-PapersOnLine, 2018, 51(21): 76-81.
[21]	BENEDEK C. Detection of soldering defects in printed circuit boards with hierarchical marked point processes [J]. Pattern Recognition Letters, 2011, 32(13): 1535- 1543.
[29]	HOU W, WEI Y, GUO J, et al. Automatic detection of welding defects using deep neural network [J]. Journal of Physics: Conference Serie, 2017, 933: 012006.
[22]	GAIDHANE V H, HOTE Y V, SINGH V. An efficient similarity measure approach for PCB surface defect detection [J]. Pattern Analysis and Applications, 2018, 21(1): 277-289.
[30]	LIN H, LI B, WANG X G, et al. Automated defect inspection of LED chip using deep convolutional neural network [J]. Journal of Intelligent Manufacturing, 2019, 30(6): 2525-2534.
[23]	ZHANG C, SHI W, LI X F, et al. Improved bare PCB defect detection approach based on deep feature learning [J]. The Journal of Engineering, 2018, 2018(16): 1415-1420.
[31]	LIN J H, YAO Y, MA L, et al. Detection of a casting defect tracked by deep convolution neural network [J]. The International Journal of Advanced Manufacturing Technology, 2018, 97(1/2/3/4): 573-581.
[24]	DAI W T, MUJEEB A, ERDT M, et al. Soldering defect detection in automatic optical inspection [J]. Advanced Engineering Informatics, 2020, 43: 101004.
[32]	NASROLLAHI M, BOLOURIAN N, HAMMAD A. Concrete surface defect detection using deep neural network based on lidar scanning [C]//CSCE Annual Conference. Laval: CSCE, 2019: CON032.
[25]	BENJDIRA B, KHURSHEED T, KOUBAA A, et al. Car detection using unmanned aerial vehicles: Comparison between faster R-CNN and YOLOv3 [C]//2019 1st International Conference on Unmanned Vehicle Systems-Oman. Muscat: IEEE, 2019: 1-6.
[33]	MEI S, WANG Y D, WEN G J. Automatic fabric defect detection with a multi-scale convolutional denoising autoencoder network model [J]. Sensors, 2018, 18(4): 1064.
[26]	LEI H W, WANG B, WU H H, et al. Defect detection for polymeric polarizer based on faster R-CNN [J]. Journal of Information Hiding and Multimedia Signal Processing, 2018, 9(6): 1414-1420.
[34]	ADIBHATLA V A, CHIH H C, HSU C C, et al. Defect detection in printed circuit boards using You-onlylook-once convolutional neural networks [J]. Electronics, 2020, 9(9): 1547.
[27]	LI Y T, HUANG H S, XIE Q S, et al. Research on a surface defect detection algorithm based on MobileNet-SSD [J]. Applied Sciences, 2018, 8(9): 1678.
[35]	ZHANG X, YANG Y H, HAN Z G, et al. Object class detection [J]. ACM Computing Surveys, 2013, 46(1): 1-53.
[28]	LI J Y, SU Z F, GENG J H, et al. Real-time detection of steel strip surface defects based on improved YOLO detection network [J]. IFAC-PapersOnLine, 2018, 51(21): 76-81.
[36]	SENGUPTA A, YE Y T, WANG R, et al. Going deeper in spiking neural networks: VGG and residual architectures [J]. Frontiers in Neuroscience, 2019, 13: 95.
[29]	HOU W, WEI Y, GUO J, et al. Automatic detection of welding defects using deep neural network [J]. Journal of Physics: Conference Serie, 2017, 933: 012006.
[37]	HENDRY, CHEN R C. Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning [J]. Image and Vision Computing, 2019, 87: 47-56.
[30]	LIN H, LI B, WANG X G, et al. Automated defect inspection of LED chip using deep convolutional neural network [J]. Journal of Intelligent Manufacturing, 2019, 30(6): 2525-2534.
[31]	LIN J H, YAO Y, MA L, et al. Detection of a casting defect tracked by deep convolution neural network [J]. The International Journal of Advanced Manufacturing Technology, 2018, 97(1/2/3/4): 573-581.
[32]	NASROLLAHI M, BOLOURIAN N, HAMMAD A. Concrete surface defect detection using deep neural network based on lidar scanning [C]//CSCE Annual Conference. Laval: CSCE, 2019: CON032.
[33]	MEI S, WANG Y D, WEN G J. Automatic fabric defect detection with a multi-scale convolutional denoising autoencoder network model [J]. Sensors, 2018, 18(4): 1064.
[34]	ADIBHATLA V A, CHIH H C, HSU C C, et al. Defect detection in printed circuit boards using You-onlylook-once convolutional neural networks [J]. Electronics, 2020, 9(9): 1547.
[35]	ZHANG X, YANG Y H, HAN Z G, et al. Object class detection [J]. ACM Computing Surveys, 2013, 46(1): 1-53.
[36]	SENGUPTA A, YE Y T, WANG R, et al. Going deeper in spiking neural networks: VGG and residual architectures [J]. Frontiers in Neuroscience, 2019, 13: 95.
[37]	HENDRY, CHEN R C. Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning [J]. Image and Vision Computing, 2019, 87: 47-56.