Spatial Temporal Correlation 3D Vehicle Detection and Tracking System with Multiple Surveillance Cameras

doi:10.1007/s12204-023-2567-1

Abstract

Abstract: Compared to 3D object detection using a single camera, multiple cameras can overcome some limitations on field-of-view, occlusion, and low detection confidence. This study employs multiple surveillance cameras and develops a cooperative 3D object detection and tracking framework by incorporating temporal and spatial information. The framework consists of a 3D vehicle detection model, cooperatively spatial-temporal relation scheme, and heuristic camera constellation method. Specifically, the proposed cross-camera association scheme combines the geometric relationship between multiple cameras and objects in corresponding detections. The spatial-temporal method is designed to associate vehicles between different points of view at a single timestamp and fulfill vehicle tracking in the time aspect. The proposed framework is evaluated based on a synthetic cooperative dataset and shows high reliability, where the cooperative perception can recall more than 66% of the trajectory instead of 11% for single-point sensing. This could contribute to full-range surveillance for intelligent transportation systems.

Key words: multi-object tracking, 3D detection, multiple sensors, cooperative perception, spatial-temporal correlation, intelligent transportation system

摘要： 相较于单个相机的三维目标检测，使用多个不同视角的相机可以克服一些缺陷，例如感知范围小、视野遮挡以及检测置信度低。本文借助多个道路监控摄像头提出了一种利用时空关联信息的联合三维目标检测与跟踪的框架，这一框架包含三维车辆检测模型、时空关联算法以及启发式的多相机安装位姿选择方法。具体来讲，提出的跨相机关联策略结合了多个相机和检测到的多个目标之间的几何关系，时空关联算法用于在某一时刻下关联不同视角的相机视野下的三维车辆完成目标检测，并且在不同时刻间关联相同车辆从而实现车辆的跟踪。我们使用仿真生成的数据集评估框架的可行性，实验结果表明联合检测达到了66%的轨迹召回率，超过了单视角检测的11%的召回率，这一框架将有助于推动智能交通系统的全覆盖。

关键词: 多目标跟踪，三维目标检测，多传感器，联合检测，时间-空间相关性，智能交通系统

CLC Number:

U121

XUE Weipeng (薛炜彭), WU Minghu (吴明虎), WANG Lin∗ (王琳). Spatial Temporal Correlation 3D Vehicle Detection and Tracking System with Multiple Surveillance Cameras[J]. J Shanghai Jiaotong Univ Sci, 2023, 28(1): 52-60.

References

[1] CHABOT F, CHAOUCH M, RABARISOA J, et al. Deep MANTA: A coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image [C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1827-1836.
[2] KUNDU A, LI Y, REHG J M. 3D-RCNN: Instance-level 3D object reconstruction via render-and-compare [C]//UT2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3559-3568.
[3] YOU Y R, WANG Y, CHAO W L, et al. PseudoLiDAR++: Accurate depth for 3D object detection in autonomous driving [EB/OL]. (2020-02-15) [2022-04-10]. https://arxiv.org/abs/1906.06310.
[4] RODDICK T, KENDALL A, CIPOLLA C. Orthographic feature transform for monocular 3D object detection [EB/OL]. (2018-11-20) [2022-04-10]. https://arxiv.org/abs/1811.08188.
[5] BRAZIL G, LIU X M. M3D-RPN: Monocular 3D region proposal network for object detection [C]//IEEE/CVF International Conference on Computer Vision (ICCV ). Seoul: IEEE, 2019: 9286-9295.
[6] ZHOU X Y, WANG D Q, KR?HENBüHL P. Objects as points [EB/OL]. (2019-04-25) [2022-04-10]. https://arxiv.org/abs/1904.07850.
[7] J?RGENSEN E, ZACH C, KAHL F. Monocular 3D object detection and box fitting trained end-to-end using intersection-over-union loss [EB/OL]. (2019-06-20)[2022-04-10]. https://arxiv.org/abs/1906.08070.
[8] WENG X S, WANG J R, HELD D, et al. 3D multi-object tracking: A baseline and new evaluation metrics [C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas: IEEE, 2020: 10359-10366.
[9] PATIL A, MALLA S, GANG H M, et al. The H3D dataset for full-surround 3D multi-object detection and tracking in crowded urban scenes [C]//International Conference on Robotics and Automation (ICRA). Montreal: IEEE, 2019: 9552-9557.
[10] OSEP A, MEHNER W, MATHIAS M, et al. Combined image- and world-space tracking in traffic scenes [C]//IEEE International Conference on Robotics and Automation. Singapore: IEEE, 2017: 1988-1995.
[11] DONG C Q, CHEN X W, DONG H B, et al. Research on intelligent vehicle infrastructure cooperative system based on Zigbee [C]//5th International Conference on Transportation Information and Safety (ICTIS). Liverpool: IEEE, 2019: 1337-1343.
[12] MAI X C, YANG M, WANG C X, et al. Multi-sensor fusion based vehicle detection and tracking method[J]. Journal of Shanghai Jiao Tong University, 2011, 45(7): 1012-1016 (in Chinese).
[13] CHEN Q, TANG S H, YANG Q, et al. Cooper: Cooperative perception for connected autonomous vehicles based on 3D point clouds [C]//IEEE 39th International Conference on Distributed Computing Systems. Dallas: IEEE, 2019: 514-524.
[14] NAPHADE M, TANG Z, CHANG M C, et al. The 2019 AI city challenge [C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 452-460.
[15] LUNA E, SANMIGUEL J C, MART′INEZ J M, et al. Online clustering-based multi-camera vehicle tracking in scenarios with overlapping FOVs [J]. Multimedia Tools and Applications, 2022, 81(5): 7063-7083.
[16] MOUSAVIAN A, ANGUELOV D, FLYNN J, et al. 3D bounding box estimation using deep learning and geometry [C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5632-5640.
[17] LAW H, DENG J. CornerNet: Detecting objects as paired keypoints [J]. International Journal of Computer Vision volume, 2020, 128: 642-656.
[18] ESTER M, KRIEGEL H P, SANDER J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise [C]//2nd International Conference on Knowledge Discovery and Data Mining (KDD-96 ). Portland: AAAI, 1996: 226-231.
[19] KALMAN R E. A new approach to linear filtering and prediction problems [J]. Journal of Basic Engineering, 1960, 82(1): 35-45.
[20] WOJKE N, BEWLEY A, PAULUS D. Simple online and realtime tracking with a deep association metric [C]//IEEE International Conference on Image Processing. Beijing: IEEE, 2017: 3645-3649.
[21] H?NEL M L, SCH?NLIEB C B. Efficient global optimization of non-differentiable, symmetric objectives for multi camera placement [J]. IEEE Sensors Journal, 2022, 22(6): 5278-5287.
[22] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite [C]//IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 3354-3361.
[23] DOSOVITSKIY A, ROS G, CODEVILLA F, et al. CARLA: An open urban driving simulator [C]//1st Conference on Robot Learning. Mountain View: PMLR, 2017: 1-16.
[24] BERNARDIN K, STIEFELHAGEN R. Evaluating multiple object tracking performance: The CLEAR MOT metrics [J]. EURASIP Journal on Image and Video Processing, 2008, 2008: 246309.
[25] YU F, WANG D Q, SHELHAMER E, et al. Deep layer aggregation [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2403-2412.