Engineering & Al for Health

Spatio-temporal Context Fusion Network for Endoscopic Ultrasound Video Segmentation

Expand
  • College of Computer Science and Technology, Donghua University, Shanghai 201620, China; 2. College of Mechanical Engineering, Donghua University, Shanghai 201620, China; 3. The Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai 200233, China

Received date: 2024-06-07

  Accepted date: 2024-08-26

  Online published: 2024-12-10

Abstract

The accurate and automatic segmentation of endoscopic ultrasound (EUS) videos is crucial for assessing the progression of submucosal tumors (SMTs) in the digestive tract and developing treatment plans. However, the dynamic nature of these videos and the rich temporal information that they contain render traditional static image processing techniques insufficient for fully extracting potential diagnostic information. To address this issue, a spatio-temporal aggregation network is proposed for the precise segmentation of EUS videos. This network comprises a spatio-temporal attention module (STAM) and a spatio-temporal context fusion module (STFM). The STAM extracts spatio-temporal features using attention mechanisms in both the temporal and spatial dimensions. The STFM synchronously integrates the spatial context from static images with the temporal context across different frames to fully explore the potential diagnostic information. The experimental results demonstrate that this method significantly improves the semantic segmentation performance of SMT videos, such as those of gastrointestinal stromal tumors and leiomyomas.

Cite this article

ZHENG Hanyao, ZHENG Hangbin, WAN Xinjian, DONG Zhixia, LI Baiyan, BAO Jinsong . Spatio-temporal Context Fusion Network for Endoscopic Ultrasound Video Segmentation[J]. Journal of Shanghai Jiaotong University(Science), 2026 , 31(5) : 1250 -1259 . DOI: 10.1007/s12204-024-2786-0

References

[1] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651.
[2] WANG R S, LEI T, CUI R X, et al. Medical image segmentation using deep learning: A survey [J]. IET Image Processing, 2022, 16(5): 1243-1267.
[3] IWASA Y, IWASHITA T, TAKEUCHI Y, et al. Automatic segmentation of pancreatic tumors using deep learning on a video image of contrast-enhanced endoscopic ultrasound [J]. Journal of Clinical Medicine, 2021, 10(16): 3589.
[4] PENG J L, WANG Y. Medical image segmentation with limited supervision: A review of deep network models [J]. IEEE Access, 2021, 9: 36827-36851.
[5] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[6] ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6230-6239.
[7] SUN G L, LIU Y, DING H H, et al. Coarse-to-fine feature mining for video semantic segmentation [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 3116-3127.
[8] CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation [M]// Computer vision – ECCV 2018. Cham: Springer, 2018: 833-851.
[9] HE J J, DENG Z Y, ZHOU L, et al. Adaptive pyramid context network for semantic segmentation [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 7511-7520.
[10] HUANG Z L, WANG X G, HUANG L C, et al. CCNet: Criss-cross attention for semantic segmentation [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 603-612.
[11] ZHANG H, DANA K, SHI J P, et al. Context encoding for semantic segmentation [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7151-7160. 
[12] ZHU Z, XU M D, BAI S, et al. Asymmetric non-local neural networks for semantic segmentation [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 593-602.
[13] GADDE R, JAMPANI V, GEHLER P V. Semantic video CNNs through representation warping [C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 4463-4472.
[14] KUNDU A, VINEET V, KOLTUN V. Feature space optimization for semantic video segmentation [C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 3168-3175.
[15] LEE S P, CHEN S C, PENG W H. GSVNET: Guided spatially-varying convolution for fast semantic segmentation on video [C]//2021 IEEE International Conference on Multimedia and Expo. Shenzhen: IEEE, 2021: 1-6.
[16] XIE E, WANG W, YU Z, et al. SegFormer: Simple and efficient design for semantic segmentation with transformers [C]// 35th Conference on Neural Information Processing Systems. Online: NIPS, 2021: 12077-12090.
[17] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// 31st Conference on Neural Information Processing Systems. Long Beach: NIPS, 2017: 1-11. 
[18] CHENG B W, MISRA I, SCHWING A G, et al. Masked-attention mask transformer for universal image segmentation [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1280-1289.
[19] OH S, KIM Y J, PARK Y T, et al. Automatic pancreatic cyst lesion segmentation on EUS images using a deep-learning approach [J]. Sensors, 2021, 22(1): 245.
[20] ZHANG J, ZHU L R, YAO L W, et al. Deep learning-based pancreas segmentation and station recognition system in EUS: Development and validation of a useful training tool (with video) [J]. Gastrointestinal Endoscopy, 2020, 92(4): 874-885.e3.
[21] MACHICADO J D, CHAO W L, CARLYN D E, et al. High performance in risk stratification of intraductal papillary mucinous neoplasms by confocal laser endomicroscopy image analysis with convolutional neural networks (with video) [J]. Gastrointestinal Endoscopy, 2021, 94(1): 78-87.e2.
[22] TANG A L, TIAN L, GAO K, et al. Contrast-enhanced harmonic endoscopic ultrasound (CH-EUS) MASTER: A novel deep learning-based system in pancreatic mass diagnosis [J]. Cancer Medicine, 2023, 12(7): 7962-7973.
[23] ZHENG H B, DONG Z X, LIU T Y, et al. Enhancing gastrointestinal submucosal tumor recognition in endoscopic ultrasonography: A novel multi-attribute guided contextual attention network [J]. Expert Systems with Applications, 2024, 242: 122725.
[24] FLEURENTIN A, MAZELLIER J P, MEYER A, et al. Automatic pancreas anatomical part detection in endoscopic ultrasound videos [J]. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 2023, 11(4): 1136-1142.
[25] HUR J, ROTH S. Joint optical flow and temporally consistent semantic segmentation [M]// Computer vision – ECCV 2016 Workshops. Cham: Springer, 2016: 163-177.
[26] SHELHAMER E, RAKELLY K, HOFFMAN J, et al. Clockwork convnets for video semantic segmentation [M]// Computer vision – ECCV 2016 Workshops. Cham: Springer, 2016: 852-868.
[27] ZHU X Z, XIONG Y W, DAI J F, et al. Deep feature flow for video recognition [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4141-4150.
[28] MAHASSENI B, TODOROVIC S, FERN A. Budget-aware deep semantic video segmentation [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2077-2086.
[29] YUAN Y, HUANG L, GUO J, et al. OCNet: Object context for semantic segmentation [J]. International Journal of Computer Vision, 2021, 129(8): 2375-2398.
Outlines

/

〈 〉