非视域成像技术研究进展

doi:10.1007/s12204-023-2686-8

摘要/Abstract

摘要： 非视域成像通过分析中继面上携带隐藏场景信息的漫反射光来恢复拐角处的隐藏物体。由于其在自动驾驶、国防、医学成像和灾后救援等领域具有巨大的应用潜力，近年来，非视域成像技术引起了国内外学者的极大关注。非视域成像相关研究主要集中在成像系统、正向模型和重建算法等方面。本文较为系统地总结了现有主动和被动场景下的非视域成像技术，并分析了非视域成像技术面临的挑战和未来发展方向。

关键词: 非视域成像, 拐弯成像, 重建算法

Abstract: Non-line-of-sight imaging recovers hidden objects around the corner by analyzing the diffuse reflection light on the relay surface that carries hidden scene information. Due to its huge application potential in the fields of autonomous driving, defense, medical imaging, and post-disaster rescue, non-line-of-sight imaging has attracted considerable attention from researchers at home and abroad, especially in recent years. The research on non-line-of-sight imaging primarily focuses on imaging systems, forward models, and reconstruction algorithms. This paper systematically summarizes the existing non-line-of-sight imaging technology in both active and passive scenes, and analyzes the challenges and future directions of non-line-of-sight imaging technology.

中图分类号:

TP391.41

. 非视域成像技术研究进展[J]. J Shanghai Jiaotong Univ Sci, 2025, 30(5): 833-854.

LIU Mengge, LIU Hao, HE Xin, JIN Shaohui, CHEN Pengyun, XU Mingliang. Research Advances on Non-Line-of-Sight Imaging Technology[J]. J Shanghai Jiaotong Univ Sci, 2025, 30(5): 833-854.

参考文献

[1] XIN S M, NOUSIAS S, KUTULAKOS K N, et al. A theory of Fermat paths for non-line-of-sight shape reconstruction [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 6793-6802.

[2] SHEN S Y, WANG Z, LIU P, et al. Non-line-of-sight imaging via neural transient fields [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(7): 2257-2268.

[3] BOUMAN K L, YE V, YEDIDIA A B, et al. Turning corners into cameras: Principles and methods [C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2289-2297.

[4] SAUNDERS C, MURRAY-BRUCE J, GOYAL V K. Computational periscopy with an ordinary digital camera [J]. Nature, 2019, 565(7740): 472-475.

[5] Maeda T, Satat G, Swedish T, et al. Recent advances in imaging around corners [DB/OL]. (2019-10-12). https://arxiv.org/abs/1910.05613

[6] FACCIO D, VELTEN A, WETZSTEIN G. Non-line-of-sight imaging [J]. Nature Reviews Physics, 2020, 2(6): 318-327.

[7] GENG R X, HU Y, CHEN Y. Recent advances on non-line-of-sight imaging: Conventional physical models, deep learning, and new scenes [J]. APSIPA Transactions on Signal and Information Processing, 2022, 11(1): e1.

[8] KIRMANI A, HUTCHISON T, DAVIS J, et al. Looking around the corner using transient imaging [C]//2009 IEEE 12th International Conference on Computer Vision. Kyoto: IEEE, 2009: 159-166.

[9] METZLER C A, HEIDE F, RANGARAJAN P, et al. Deep-inverse correlography: Towards real-time high-resolution non-line-of-sight imaging [J]. Optica, 2020, 7(1): 63.

[10] LEI X, HE L Y, TAN Y X, et al. Direct object recognition without line-of-sight using optical coherence [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 11729-11738.

[11] SMITH B M, O'TOOLE M, GUPTA M. Tracking multiple objects outside the line of sight using speckle imaging [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6258-6266.

[12] XU F H, SHULKIND G, THRAMPOULIDIS C, et al. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging [J]. Optics Express, 2018, 26(8): 9945.

[13] CHEN W Z, DANEAU S, BROSSEAU C, et al. Steady-state non-line-of-sight imaging [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 6783-6792.

[14] TANCIK M, SATAT G, RASKAR R. Flash photography for data-driven hidden scene recovery [DB/OL]. (2018-10-27). https://arxiv.org/abs/1810.11710

[15] ZHOU C, WANG C Y, LIU Z W. Non-line-of-sight imaging off a Phong surface through deep learning [DB/OL]. (2020-04-30). https://arxiv.org/abs/2005.00007

[16] CAO Y P, LIANG R, ZHU W B, et al. Dynamic-excitation-based steady-state non-line-of-sight imaging via multi-branch convolutional neural network [J]. Optics and Lasers in Engineering, 2023, 161: 107369.

[17] AHN B, DAVE A, VEERARAGHAVAN A, et al. Convolutional approximations to the general non-line-of-sight imaging operator [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: 2019: 7888-7898.

[18] VELTEN A, WILLWACHER T, GUPTA O, et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging [J]. Nature Communications, 2012, 3: 745.

[19] O’TOOLE M, LINDELL D B, WETZSTEIN G. Confocal non-line-of-sight imaging based on the light-cone transform [J]. Nature, 2018, 555(7696): 338-341.

[20] HEIDE F, O’TOOLE M, ZANG K, et al. Non-line-of-sight imaging with partial occluders and surface normals [J]. ACM Transactions on Graphics, 2019, 38(3): 22.

[21] GUPTA O, WILLWACHER T, VELTEN A, et al. Reconstruction of hidden 3D shapes using diffuse reflections [J]. Optics Express, 2012, 20(17): 19096.

[22] BUTTAFAVA M, ZEMAN J, TOSI A, et al. Non-line-of-sight imaging using a time-gated single photon avalanche diode [J]. Optics Express, 2015, 23(16): 20997-21011.

[23] ARELLANO V, GUTIERREZ D, JARABO A. Fast back-projection for non-line of sight reconstruction [J]. Optics Express, 2017, 25(10): 11574.

[24] LA MANNA M, KINE F, BREITBACH E, et al. Error backprojection algorithms for non-line-of-sight imaging [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(7): 1615-1626.

[25] YOUNG S I, LINDELL D B, GIROD B, et al. Non-line-of-sight surface reconstruction using the directional light-cone transform [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1404-1413.

[26] HEIDE F, XIAO L, HEIDRICH W, et al. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors [C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 3222-3229.

[27] TSAI C Y, KUTULAKOS K N, NARASIMHAN S G, et al. The geometry of first-returning photons for non-line-of-sight imaging [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2336-2344.

[28] TSAI C Y, SANKARANARAYANAN A C, GKIOULEKAS I. Beyond volumetric albedo—a surface optimization framework for non-line-of-sight imaging [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 1545-1555.

[29] PEDIREDLA A K, BUTTAFAVA M, TOSI A, et al. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner [C]//2017 IEEE International Conference on Computational Photography. Stanford: IEEE, 2017: 1-12.

[30] ISERINGHAUSEN J, HULLIN M B. Non-line-of-sight reconstruction using efficient transient rendering [J]. ACM Transactions on Graphics, 2020, 39(1): 1-14.

[31] LINDELL D B, WETZSTEIN G, O'TOOLE M. Wave-based non-line-of-sight imaging using fast f-k migration [J]. ACM Transactions on Graphics, 2019, 38(4): 116.

[32] LIU X C, GUILLÉN I, LA MANNA M, et al. Non-line-of-sight imaging using phasor-field virtual wave optics [J]. Nature, 2019, 572(7771): 620-623.

[33] LIU X C, BAUER S, VELTEN A. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems [J]. Nature Communications, 2020, 11: 1645.

[34] REZA S A, LA MANNA M, BAUER S, et al. Phasor field waves: A Huygens-like light transport model for non-line-of-sight imaging applications [J]. Optics Express, 2019, 27(20): 29380.

[35] NAM J H, BRANDT E, BAUER S, et al. Low-latency time-of-flight non-line-of-sight imaging at 5 frames per second [J]. Nature Communications, 2021, 12: 6526.

[36] GRAU CHOPITE J, HULLIN M B, WAND M, et al. Deep non-line-of-sight reconstruction [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 957-966.

[37] CARAMAZZA P, BOCCOLINI A, BUSCHEK D, et al. Neural network identification of people hidden from view with a single-pixel, single-photon detector [J]. Scientific Reports, 2018, 8: 11945.

[38] PENG J Y, MU F Z, NAM J, et al. Towards non-line-of-sight photography [DB/OL]. (2020-04-30). https://arxiv.org/abs/2005.00007

[39] CHEN W Z, WEI F Y, KUTULAKOS K N, et al. Learned feature embeddings for non-line-of-sight imaging and recognition [J]. ACM Transactions on Graphics, 2020, 39(6): 230.

[40] ISOGAWA M, YUAN Y, O'TOOLE M, et al. Optical non-line-of-sight physics-based 3D human pose estimation [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 7011-7020.

[41] MU F Z, MO S C, PENG J Y, et al. Physics to the rescue: Deep non-line-of-sight reconstruction for high-speed imaging [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. https://doi.org/10.1109/TPAMI.2022.3203383

[42] GRAU J, PLACK M, HAEHN P, et al. Occlusion fields: An implicit representation for non-line-of-sight surface reconstruction [DB/OL]. (2022-03-16). https://arxiv.org/abs/2203.08657

[43] ISOGAWA M, CHAN D, YUAN Y, et al. Efficient non-line-of-sight imaging from transient sinograms[M]//European conference on computer vision. Cham: Springer, 2020: 193-208.

[44] PEDIREDLA A, DAVE A, VEERARAGHAVAN A. SNLOS: non-line-of-sight scanning through temporal focusing [C]//2019 IEEE International Conference on Computational Photography. Tokyo: IEEE, 2019: 1-13.

[45] YANG W Q, ZHANG C, JIANG W J, et al. None-line-of-sight imaging enhanced with spatial multiplexing [J]. Optics Express, 2022, 30(4): 5855.

[46] SEIDEL S W, MA Y T, MURRAY-BRUCE J, et al. Corner occluder computational periscopy: Estimating a hidden scene from a single photograph [C]//2019 IEEE International Conference on Computational Photography. Tokyo: IEEE, 2019: 1-9.

[47] SEIDEL S W, MURRAY-BRUCE J, MA Y T, et al. Two-dimensional non-line-of-sight scene estimation from a single edge occluder [J]. IEEE Transactions on Computational Imaging, 2021, 7: 58-72.

[48] MAEDA T, WANG Y Q, RASKAR R, et al. Thermal non-line-of-sight imaging [C]//2019 IEEE International Conference on Computational Photography. Tokyo: IEEE, 2019: 1-11.

[49] KAGA M, KUSHIDA T, TAKATANI T, et al. Thermal non-line-of-sight imaging from specular and diffuse reflections [J]. IPSJ Transactions on Computer Vision and Applications, 2019, 11(1): 1-6.

[50] BARADAD M, YE V, YEDIDIA A B, et al. Inferring light fields from shadows [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6267-6275.

[51] YEDIDIA A B, BARADAD M, THRAMPOULIDIS C, et al. Using unknown occluders to recover hidden scenes [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 12223-12231.

[52] LIN D, HASHEMI C, LEGER J R. Passive non-line-of-sight imaging using plenoptic information [J]. Journal of the Optical Society of America A, 2020, 37(4): 540-551.

[53] SASAKI T, LEGER J R. Non-line-of-sight object location estimation from scattered light using plenoptic data [J]. Journal of the Optical Society of America A, 2021, 38(2): 211-228.

[54] BATARSEH M, SUKHOV S, SHEN Z, et al. Passive sensing around the corner using spatial coherence [J]. Nature Communications, 2018, 9: 3629.

[55] BECKUS A, TAMASAN A, ATIA G K. Multi-modal non-line-of-sight passive imaging [J]. IEEE Transactions on Image Processing, 2019, 28(7): 3372-3382.

[56] BOGER-LOMBARD J, KATZ O. Passive optical time-of-flight for non line-of-sight localization [J]. Nature Communications, 2019, 10: 3343.

[57] DIVITT S, GARDNER D F, WATNIK A T. Imaging around corners in the mid-infrared using speckle correlations [J]. Optics Express, 2020, 28(8): 11051.

[58] HASSAN B. Polarization-informed non-line-of-sight imaging on diffuse surfaces[D]. Los Angeles: University of California, 2019.

[59] TANAKA K, MUKAIGAWA Y, KADAMBI A. Polarized non-line-of-sight imaging [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2133-2142.

[60] Aittala M, Sharma P, Murmann L, et al. Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization [M]// Advances in neural information processing systems 32. Red Hook: Curran Associates Inc., 2019: 14324-14334.

[61] WU H Z, LIU S P, MENG X F, et al. Non-line-of-sight imaging based on an untrained deep decoder network [J]. Optics Letters, 2022, 47(19): 5056.

[62] YU T Y, QIAO M, LIU H L, et al. Non-line-of-sight imaging through deep learning [J]. Acta Optica Sinica, 2019, 39(7): 0711002 (in Chainese).

[63] WANG Y Y, ZHANG Y Q, HUANG M Y, et al. Accurate but fragile passive non-line-of-sight recognition [J]. Communications Physics, 2021, 4: 88.

[64] HE J H, WU S K, WEI R, et al. Non-line-of-sight imaging and tracking of moving objects based on deep learning [J]. Optics Express, 2022, 30(10): 16758-16772.

[65] GENG R X, HU Y, LU Z, et al. Passive non-line-of-sight imaging using optimal transport [J]. IEEE Transactions on Image Processing, 2022, 31: 110-124.

[66] WU C, LIU J J, HUANG X, et al. Non–line-of-sight imaging over 1.43 km [J]. Proceedings of the National Academy of Sciences of the United States of America, 2021, 118(10): e2024468118.

[67] METZLER C A, LINDELL D B, WETZSTEIN G. Keyhole imaging: Non-line-of-sight imaging and tracking of moving objects along a single optical path [J]. IEEE Transactions on Computational Imaging, 2021, 7: 1-12.

[68] CHAN S S, WARBURTON R E, GARIEPY G, et al. Non-line-of-sight tracking of people at long range [J]. Optics Express, 2017, 25(9): 10109-10117.

[69] ZHANG M L, SHI Y Y, SHENG W, et al. Non-line-of-sight imaging based on Archimedean spiral scanning [J]. Optics Communications, 2023, 537: 129450.

[70] YE J T, HUANG X, LI Z P, et al. Compressed sensing for active non-line-of-sight imaging [J]. Optics Express, 2021, 29(2): 1749.

[71] DING R, YE J T, GAO Q F, et al. Curvature regularization for non-line-of-sight imaging from under-sampled data [DB/OL]. (2023-01-01). https://arxiv.org/abs/2301.00406

[72] ZHU S Y, SUA Y M, BU T, et al. Compressive non-line-of-sight imaging with deep learning [J]. Physical Review Applied, 2023, 19(3): 034090.

[73] O'TOOLE M, LINDELL D B, WETZSTEIN G. Real-time non-line-of-sight imaging [C]//ACM SIGGRAPH 2018 Emerging Technologies. Vancouver: ACM, 2018: 1-2.

[74] PEI C Q, ZHANG A K, DENG Y E, et al. Dynamic non-line-of-sight imaging system based on the optimization of point spread functions [J]. Optics Express, 2021, 29(20): 32349-32364.

[75] LIAO Z P, JIANG D Y, LIU X C, et al. FPGA accelerator for real-time non-line-of-sight imaging [J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2022, 69(2): 721-734.

[76] PLACK M, CALLENBERG C, SCHNEIDER M, et al. Fast differentiable transient rendering for non-line-of-sight reconstruction [C]//2023 IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2023: 3066-3075.

[77] FUJIMURA Y, KUSHIDA T, FUNATOMI T, et al. NLOS-NeuS: Non-line-of-sight neural implicit surface [DB/OL]. (2023-03-22). https://arxiv.org/abs/2303.12280