J Shanghai Jiaotong Univ Sci

Journal of Shanghai Jiaotong University(Science) >

2026 , Vol. 31 >Issue 3: 646 - 659

DOI: https://doi.org/10.1007/s12204-026-2931-z

Celebrating 30 Years

Deep Residual Learning-Based AprilTag Surgical Instrument Localization and Tracking

Expand
  • 1. School of Computer Science and Engineering, Central South University, Changsha 410017, China; 2. Department of Rehabilitation, The Second Xiangya Hospital, Central South University, Changsha 410011, China; 3. Clinical and Translational Research Center for 3D Printing Technology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200011, China; 4. School of Computer Science and Engineering, Jishou University, Jishou 416000, Hunan, China

Received date: 2025-11-26

  Revised date: 2026-02-14

  Accepted date: 2026-03-24

  Online published: 2026-06-11

Fold

Abstract

Surgical navigation technology provides surgeons with precise guidance through real-time tracking of the spatial pose of surgical instruments and serves as a crucial technical foundation for modern precision medicine. As a visual reference marker system, AprilTag demonstrates the advantages of low cost and flexible deployment for surgical instrument localization. However, traditional geometric constraint-based pose estimation methods face challenges such as cumulative detection errors and environmental interference sensitivity in complex surgical environments, making it difficult to meet the requirements of high-precision surgical navigation. This study proposes a novel AprilTag pose tracking method that integrates geometric and deep learning approaches. By constructing a deep residual learning network, the method learns the influence patterns of image features on pose errors to achieve intelligent correction of traditional geometric methods. The approach employs ResNet18 to extract deep image features, encodes the AprilTag’s pose information, and utilizes a dual-branch fusion network to learn the residual error between geometric predictions and actual instrument pose. A weighted loss function is designed to address rotation and translation errors. Experimental results on real surgical instrument datasets show that compared to traditional geometric methods, this approach reduces the average position error by 48.44% and the average rotation error by 9.66%. Comparative tests with other visual positioning methods further demonstrate the method’s superior comprehensive performance in complex scenarios.

Cite this article

Zhong Jihao, Liao Shenghui, Jiang Wenbo, Li Jianfeng, Liu Lihong, Kui Xiaoyan . Deep Residual Learning-Based AprilTag Surgical Instrument Localization and Tracking[J]. Journal of Shanghai Jiaotong University(Science), 2026 , 31(3) : 646 -659 . DOI: 10.1007/s12204-026-2931-z

References

[1] Cleary K, Peters T M. Image-guided interventions: Technology review and clinical applications [J]. Annual Review of Biomedical Engineering, 2010, 12: 119-142.
[2] Mezger U, Jendrewski C, Bartels M. Navigation in surgery [J]. Langenbeck’s Archives of Surgery, 2013, 398(4): 501-514.
[3] Willems P W A, van der Sprenkel J W B, Tulleken C A F, et al. Neuronavigation and surgery of intracerebral tumours [J]. Journal of Neurology, 2006, 253(9): 1123-1136.
[4] Franz A M, Haidegger T, Birkfellner W, et al. Electromagnetic tracking in medicine: A review of technology, validation, and applications [J]. IEEE Transactions on Medical Imaging, 2014, 33(8): 1702-1725.
[5] Xu L S, Zhang H R, Wang J L, et al. Information loss challenges in surgical navigation systems: From information fusion to AI-based approaches [J]. Information Fusion, 2023, 92: 13-36.
[6] Yang L J, Etsuko K. Review on vision-based tracking in surgical navigation [J]. IET Cyber-Systems and Robotics, 2020, 2(3): 107-121.
[7] Olson E. AprilTag: A robust and flexible visual fiducial system [C]//2011 IEEE International Conference on Robotics and Automation. Shanghai: IEEE, 2011: 3400-3407.
[8] Wang J, Olson E. AprilTag 2: Efficient and robust fiducial detection [C]//2016 IEEE/RSJ International Conference on Intelligent Robots and Systems. Daejeon: IEEE, 2016: 4193-4198.
[9] Krogius M, Haggenmiller A, Olson E. Flexible layouts for fiducial tags [C]//2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Macau: IEEE, 2020: 1898-1903.
[10] Mantha B R K, Garcia de Soto B. Investigating the fiducial marker network characteristics for autonomous mobile indoor robot navigation using ROS and gazebo [J]. Journal of Construction Engineering and Management, 2022, 148(10): 04022115.
[11] Li W L, Xie H, Zhang G, et al. Hand-eye calibration in visually-guided robot grinding [J]. IEEE Transactions on Cybernetics, 2016, 46(11): 2634-2642.
[12] Lepetit V, Moreno-Noguer F, Fua P. EPnP: An accurate O(n) solution to the PnP problem [J]. International Journal of Computer Vision, 2009, 81(2): 155-166.
[13] Kneip L, Scaramuzza D, Siegwart R. A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation [C]//CVPR 2011. Colorado Springs: IEEE, 2011: 2969-2976.
[14] Xiang Y, Schmidt T, Narayanan V, et al. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes [PP/OL]. V3. arXiv (2018-05-26) [2025-11-23]. https://doi.org/10.48550/arXiv.1711.00199.
[15] Tekin B, Sinha S N, Fua P. Real-time seamless single shot 6D object pose prediction [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 292-301.
[16] Pavlakos G, Zhou X W, Chan A, et al. 6-DoF object pose from semantic keypoints [C]//2017 IEEE International Conference on Robotics and Automation. Singapore: IEEE, 2017: 2011-2018.
[17] Hu Y L, Hugonot J, Fua P, et al. Segmentation-driven 6D object pose estimation [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 3380-3389.
[18] Tang G Q, Yang T F, Yang Y, et al. Relative localization and dynamic tracking of underwater robots based on 3D-AprilTag [J]. Journal of Marine Science and Engineering, 2025, 13(5): 833.
[19] Maier-Hein L, Vedula S S, Speidel S, et al. Surgical data science for next-generation interventions [J]. Nature Biomedical Engineering, 2017, 1(9): 691-696.
[20] Bodenstedt S, Allan M, Agustinos A, et al. Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery [PP/OL]. arXiv (2018-05-07) [2025-11-23]. https://doi.org/10.48550/arXiv.1805.02475.
[21] Du X F, Kurmann T, Chang P L, et al. Articulated multi-instrument 2-D pose estimation using fully convolutional networks [J]. IEEE Transactions on Medical Imaging, 2018, 37(5): 1276-1287.
[22] Zhang Z. A flexible new technique for camera calibration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330-1334.
[23] Marchand E, Spindler F, Chaumette F. ViSP for visual servoing: A generic software platform with a wide class of robot control skills [J]. IEEE Robotics & Automation Magazine, 2005, 12(4): 40-52.
[24] Markley F L, Cheng Y, Crassidis J L, et al. Averaging quaternions [J]. Journal of Guidance, Control, and Dynamics, 2007, 30(4): 1193-1197.
[25] Shoemake K. Animating rotation with quaternion curves [C]//Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques. San Francisco: ACM, 1985: 245-254.
[26] Yosinski J, Clune J, Bengio Y, et al. How transferable are features in deep neural networks? [C]// 28th International Conference on Neural Information Processing Systems. Montreal: NIPS, 2014: 3320-3328.
[27] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks [J]. Communications of the ACM, 2017, 60(6): 84-90.
[28] Murray R M, Li Z X, Sastry S. A mathematical introduction to robotic manipulation [M]. Boca Raton: CRC Press, 2017.
[29] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift [C]// 32nd International Conference on Machine Learning. Lille: PMLR, 2015: 448-456. 
[30] Srivastava N, Hinton G E, Krizhevsky A, et al. Dropout: A simple way to prevent neural networks from overfitting [J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
[31] Zhou X Y, Wang D Q, Krähenbühl P. Objects as points [PP/OL]. V2. arXiv (2019-04-25) [2025-11-23]. https://doi.org/10.48550/arXiv.1904.07850.
[32] LeCun Y, Bengio Y, Hinton G. Deep learning [J]. Nature, 2015, 521(7553): 436-444.
[33] Kingma D P, Ba J. Adam: A method for stochastic optimization [PP/OL]. V9. arXiv (2017-01-30) [2025-11-23]. https://doi.org/10.48550/arXiv.1412.6980.
[34] Dam E B, Koch M, Lillholm M. Quaternions, interpolation and animation[M]. Copenhagen: Datalogisk Institut, Københavns Universitet, 1998.
[35] Shorten C, Khoshgoftaar T M. A survey on image data augmentation for deep learning [J]. Journal of Big Data, 2019, 6(1): 60.
[36] Guo W, Wu J, Chen Z, et al. SurgTrack: CAD-free 3D tracking of real-world surgical instruments [M]// Medical image computing and computer assisted intervention – MICCAI 2024 Workshops. Cham: Springer, 2025: 168-177.
[37] Wen B W, Bekris K. BundleTrack: 6D pose tracking for novel objects without instance or category-level 3D models [C]//2021 IEEE/RSJ International Conference on Intelligent Robots and Systems. Prague: IEEE, 2021: 8067-8074.
[38] Sun J M, Wang Z H, Zhang S Y, et al. OnePose: One-shot object pose estimation without CAD models [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 6815-6824.

Options
Outlines

/

〈 〉