CT-MFENet: Context Transformer and Multi-Scale Feature Extraction Network via Global-Local Features Fusion for Retinal Vessels Segmentation

doi:10.1007/s12204-024-2748-6

Abstract

Abstract: Segmentation of the retinal vessels in the fundus is crucial for diagnosing ocular diseases. Retinal vessel images often suffer from category imbalance and large scale variations. This ultimately results in incomplete vessel segmentation and poor continuity. In this study, we propose CT-MFENet to address the aforementioned issues. First, the use of context transformer (CT) allows for the integration of contextual feature information, which helps establish the connection between pixels and solve the problem of incomplete vessel continuity. Second, multi-scale dense residual networks are used instead of traditional CNN to address the issue of inadequate local feature extraction when the model encounters vessels at multiple scales. In the decoding stage, we introduce a local-global fusion module. It enhances the localization of vascular information and reduces the semantic gap between high- and low-level features. To address the class imbalance in retinal images, we propose a hybrid loss function that enhances the segmentation ability of the model for topological structures. We conducted experiments on the publicly available DRIVE, CHASEDB1, STARE, and IOSTAR datasets. The experimental results show that our CT-MFENet performs better than most existing methods, including the baseline U-Net.

Key words: retinal vessel segmentation, context transformer (CT), multi-scale dense residual, hybrid loss function, global-local fusion

摘要： 眼底视网膜血管的分割对于诊断眼部疾病至关重要。视网膜血管图像分割通常受到类别不平衡和血管尺度变化大的影响。这最终导致分割血管的不完整和连续性较差。本研究中，提出了CT-MFENet来解决上述问题。首先，使用上下文Transformer（CT）来整合上下文特征信息，这有助于像素间的远程建模，从而解决了血管连续性不完整的问题。其次，使用多尺度密集残差模块代替传统的CNN，以解决模型遇到多尺度血管时局部特征提取能力不足的问题。在解码阶段，引入的局部-全局融合模块增强了局部血管信息，并减小了高低级别特征之间的语义间隙。为了解决视网膜图像中的类别不平衡问题，提出一种混合损失函数，增强了模型对拓扑结构的分割能力。在公开的DRIVE、CHASEDB1、STARE和IOSTAR数据集上进行了实验。实验结果表明，提出的CT-MFENet在性能上优于大多数现有方法，包括基模型U-Net。

关键词: 视网膜血管分割，上下文Transformer，多尺度稠密残差，混合损失函数，全局-局部融合

CLC Number:

TP391
R319

Shao Dangguo, Yang Yuanbiao, Ma Lei, Yi Sanli. CT-MFENet: Context Transformer and Multi-Scale Feature Extraction Network via Global-Local Features Fusion for Retinal Vessels Segmentation[J]. J Shanghai Jiaotong Univ Sci, 2025, 30(4): 668-682.

References

[1] ABRAMOFF M D, GARVIN M K, SONKA M. Retinal imaging and image analysis [J]. IEEE Reviews in Biomedical Engineering, 2010, 3: 169-208. [2] SHENG B, LI P, MO S, et al. Retinal vessel segmentation using minimum spanning superpixel tree detector [J]. IEEE Trans Cybern, 2019, 49(7): 2707-2719. [3] JIN Q G, MENG Z P, PHAM T D, et al. DUNet: A deformable network for retinal vessel segmentation [J]. Knowledge-Based Systems, 2019, 178: 149-162. [4] MOU L, CHEN L, CHENG J, et al. Dense dilated network with probability regularized walk for vessel detection [J]. IEEE Trans Med Imaging, 2020, 39(5): 1392-1403. [5] WU H, WANG W, ZHONG J, et al. SCS-net: A scale and context sensitive network for retinal vessel segmentation [J]. Med Image Anal, 2021, 70: 102025. [6] GALDRAN A, ANJOS A, DOLZ J, et al. State-of-the-art retinal vessel segmentation with minimalistic models [J]. Scientific Reports, 2022, 12(1): 6174. [7] CHAUDHURI S, CHATTERJEE S, KATZ N, et al. Detection of blood vessels in retinal images using two-dimensional matched filters [J]. IEEE Transactions on Medical Imaging, 1989, 8(3): 263-269. [8] CAN A L, SHEN H, TURNER J N, et al. Rapid automated tracing and feature extraction from retinal fundus images using direct exploratory algorithms [J]. IEEE Transactions on Information Technology in Biomedicine, 1999, 3(2): 125-138. [9] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// 31st Conference on Neural Information Processing Systems. Long Beach: NIPS, 2017: 1-11. [10] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale [DB/OL]. (2020-10-22). https://arxiv.org/abs/2010.11929 [11] RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation [M]// Medical image computing and computer-assisted intervention – MICCAI 2015. Cham: Springer, 2015: 234-241. [12] CHEN J N, LU Y Y, YU Q H, et al. TransUNet: Transformers make strong encoders for medical image segmentation [DB/OL]. (2021-02-08). http://arxiv.org/abs/2102.04306 [13] WU Y C, XIA Y, SONG Y, et al. Multiscale network followed network model for retinal vessel segmentation [M]//Medical image computing and computer assisted intervention – MICCAI 2018. Cham: Springer, 2018: 119-126. [14] OKTAY O, SCHLEMPER J, LE FOLGOC L, et al. Attention U-net: Learning where to look for the pancreas [DB/OL]. (2018-04-11). http://arxiv.org/abs/1804.03999 [15] YAN Z Q, YANG X, CHENG K T. Joint segment-level and pixel-wise losses for deep learning based retinal vessel segmentation [J]. IEEE Transactions on Biomedical Engineering, 2018, 65(9): 1912-1923. [16] LI X, CHEN H, QI X, et al. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes [J]. IEEE Trans Medical Imaging, 2018, 37(12): 2663-2674. [17] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition [C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778. [18] GHIASI G, LIN T Y, LE Q V. DropBlock: A regularization method for convolutional networks [C]// 32nd International Conference on Neural Information Processing Systems. Montréal: NIPS, 2018: 10750-10760. [19] HU J, SHEN L, SUN G. Squeeze-and-excitation networks [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141. [20] STAAL J, ABRAMOFF M D, NIEMEIJER M, et al. Ridge-based vessel segmentation in color images of the retina [J]. IEEE Transactions on Medical Imaging, 2004, 23(4): 501-509. [21] OWEN C G, RUDNICKA A R, MULLEN R, et al. Measuring retinal vessel tortuosity in 10-year-old children: Validation of the computer-assisted image analysis of the retina (CAIAR) program [J]. Investigative Opthalmology & Visual Science, 2009, 50(5): 2004. [22] WANG B, QIU S, HE H G. Dual encoding U-net for retinal vessel segmentation [M]// medical image computing and computer assisted intervention – MICCAI 2019. Cham: Springer, 2019: 84-92. [23] HOOVER A D, KOUZNETSOVA V, GOLDBAUM M. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response [J]. IEEE Transactions on Medical Imaging, 2000, 19(3): 203-210. [24] ZHANG J, DASHTBOZORG B, BEKKERS E, et al. Robust retinal vessel segmentation via locally adaptive derivative frames in orientation scores [J]. IEEE Transactions on Medical Imaging, 2016, 35(12): 2631-2644. [25] ZUIDERVELD K. Contrast limited adaptive histogram equalization [M]//Graphics gems. Amsterdam: Elsevier, 1994: 474-485. [26] WANG S H, LI L, ZHUANG X H. AttU-NET: Attention U-net for brain tumor segmentation [M]// Brainlesion: Glioma, multiple sclerosis, stroke and traumatic brain injuries. Cham: Springer, 2022: 302-311. [27] LI L Z, VERMA M, NAKASHIMA Y, et al. IterNet: Retinal image segmentation utilizing structural redundancy in vessel networks [C]//2020 IEEE Winter Conference on Applications of Computer Vision. Snowmass: IEEE, 2020: 3645-3654. [28] HUANG H M, LIN L F, TONG R F, et al. UNet 3: A full-scale connected UNet for medical image segmentation [C]// 2020 IEEE International Conference on Acoustics, Speech and Signal Processing. Barcelona: IEEE, 2020: 1055-1059. [29] SAHA TCHINDA B, TCHIOTSOP D, NOUBOM M, et al. Retinal blood vessels segmentation using classical edge detection filters and the neural network [J]. Informatics in Medicine Unlocked, 2021, 23: 100521. [30] JIANG Y, LIU W, WU C, et al. Multi-scale and multi-branch convolutional neural network for retinal image segmentation [J]. Symmetry, 2021, 13(3): 365. [31] ZHANG Y, HE M, CHEN Z, et al. Bridge-Net: Context-involved U-net with patch-based loss weight mapping for retinal blood vessel segmentation [J]. Expert Systems with Applications, 2022, 195: 116526. [32] LI J Y, GAO G, LIU Y H, et al. MAGF-Net: A multiscale attention-guided fusion network for retinal vessel segmentation [J]. Measurement, 2023, 206: 112316. [33] LI Z, LI B Y, ZHANG J, et al. MFCTrans-net: A multi-scale fusion and channel transformer net for retinal vessel segmentation [C]//Fourteenth International Conference on Graphics and Image Processing. Nanjing: SPIE, 2023: 537-545. [34] LI X, JIANG Y C, LI M L, et al. Lightweight attention convolutional neural network for retinal vessel image segmentation [J]. IEEE Transactions on Industrial Informatics, 2021, 17(3): 1958-1967.