The classification of the defects in welding applications based on the metallographic structure images plays an important part in industrial welding quality inspections. In order to improve the classification performance of defects in the weld metallographic structure images with a small sample dataset available (the amount of samples being less than 30), a Poisson fusion method is used for data augmentation of the defect images and the ResNet18_PRO network is proposed. Both of the methods notably improve the defects classification performance. During data augmentation, the defect area is extracted from original defect samples via digital image processing, and the defect area is fused with normal samples by the Poisson fusion method to generate new defect samples, thus increasing the number of defect samples. The model in this paper is improved based on the ResNet18 network. The downsampling structure is improved to reduce the information loss in the original downsampling structure, and an improved space pyramid pooling structure is added at the end of the network to integrate multi-scale feature information. The classification performance before and after data augmentation is compared by different classification models, which verifies the significant effect of the data augmentation on the classification performance. Meanwhile, the ablation experiment of the ResNet18_PRO is conducted to verify the effectiveness of the improved network structure and the training strategy. It is found that the average classification accuracy of ResNet18_PRO reaches 98.83% and the average F₁-score reaches 98.76%, which greatly improves the classification accuracy of metallographic structure defects. Finally, the network is trained and tested with another industrial defect dataset and obtains good classification results. These results show that the proposed network has a good robustness and practical application value.