Intelligent Analysis of Abnormal Vehicle Behavior Based on a Digital Twin

doi:10.1007/s12204-021-2348-7

Abstract

Abstract: Analyzing a vehicle’s abnormal behavior in surveillance videos is a challenging field, mainly due to the wide variety of anomaly cases and the complexity of surveillance videos. In this study, a novel intelligent vehicle behavior analysis framework based on a digital twin is proposed. First, detecting vehicles based on deep learning is implemented, and Kalman filtering and feature matching are used to track vehicles. Subsequently, the tracked vehicle is mapped to a digital-twin virtual scene developed in the Unity game engine, and each vehicle’s behavior is tested according to the customized detection conditions set up in the scene. The stored behavior data can be used to reconstruct the scene again in Unity for a secondary analysis. The experimental results using real videos from traffic cameras illustrate that the detection rate of the proposed framework is close to that of the state-of-the-art abnormal event detection systems. In addition, the implementation and analysis process show the usability, generalization, and effectiveness of the proposed framework.

Key words: digital twin, deep learning, vehicle detection, abnormal behavior

CLC Number:

TP 391.4

LI Lin (李霖), HU Zeyu(胡泽宇), YANG Xubo (杨旭波). Intelligent Analysis of Abnormal Vehicle Behavior Based on a Digital Twin[J]. J Shanghai Jiaotong Univ Sci, 2021, 26(5): 587-597.

References 23

[1]	JIANG F, WU Y, KATSAGGELOS A K. Abnormal event detection from surveillance video by dynamic hierarchical clustering [C]//2007 IEEE International Conference on Image Processing. San Antonio, TX: IEEE, 2007: V145-V148.
[2]	WANG J Q, LI H P. Intelligent analysis of vehicle illegal behavior based on deep learning algorithm [J]. Journal of Shanghai Ship and Shipping Research Insti-tute, 2019, 42(4): 49-54 (in Chinese). [3] FULLERA,FAN Z, DAYC,et al. Digital twin: En-abling technologies, challenges and open research [J]. IEEE Access, 2020, 8: 108952-108971.
[4]	KUMAR S A P, MADHUMATHI R, CHELLIAH P R, et al. A novel digital twin-centric approach for driver intention prediction and tra?c congestion avoidance [J]. Journal of Reliable Intelligent Environments, 2018, 4(4): 199-209.
[5]	HAM Y, KIM J. Participatory sensing and digital twin City: Updating virtual City models for enhanced risk-informed decision-making [J]. Journal of Management in Engineering, 2020, 36(3): 04020005.
[6]	JOCHER G, NISHIMURA K, MINEEVA T, et al. YOLOv5 (2020) [EB/OL]. (2020-07-10) [2021-01-30]. https://github.com/ultralytics/YOLOv5.
[7]	WANG C Y, MARK LIAO H Y, WU Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Work-shops (CVPRW ). Seattle, WA: IEEE, 2020: 1571-1580.
[8]	LIN T Y, DOLL ′AR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]//2017 IEEE Conference on Computer Vision and Pattern Recogni-tion (CVPR). Honolulu, HI: IEEE, 2017: 936-944.
[9]	WANG W H, XIE E Z, SONG X G, et al. E?cient and accurate arbitrary-shaped text detection with pixel aggregation network [C]//2019 IEEE/CVF Interna-tional Conference on Computer Vision (ICCV ). Seoul: IEEE, 2019: 8439-8448.
[10]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: Common objects in context [C]//Computer Vision–ECCV 2014. Cham: Springer, 2014: 740-755.
[11]	WEN L Y, DU D W, CAI Z W, et al. UA-DETRAC: A new benchmark and protocol for multi-object de-tection and tracking [J]. Computer Vision and Image Understanding, 2020, 193: 102907.
[12]	PUNN N S, SONBHADRA S K, AGARWAL S, et al. Monitoring COVID-19 social distancing with person detection and tracking via ?ne-tuned YOLO v3 and Deepsort techniques [EB/OL]. [2021-01-30]. https://arxiv.org/pdf/2005.01385.pdf.
[13]	MIKHAIL E M, BETHEL J S, MCGLONE J C. In-troduction to modern photogrammetry [M]. New York: John Wiley, 2001. [14] HUSSEIN A, GARC′IA F, OLAVERRI-MONREAL C. ROS and unity based framework for intelligent vehi-cles control and simulation [C]//2018 IEEE Interna-tional Conference on Vehicular Electronics and Safety (ICVES ). Madrid: IEEE, 2018: 1-6.
[15]	ZHAO C H, LI Y W, SHI X, et al. Recognition of abnormal vehicle behaviors using combined features [C]//19th COTA International Conference of Trans-portation Professionals. Reston, VA: ASCE, 2019: 2180-2186.
[16]	CHEN Z J, WU C Z, HUANG Z, et al. Dangerous driving behavior detection using video-extracted ve-hicle trajectory histograms [J]. Journal of Intelligent Transportation Systems, 2017, 21(5): 409-421.
[17]	DOSHI K, YILMAZ Y. Fast unsupervised anomaly de-tection in tra?c videos [C]//2020 IEEE/CVF Con-ference on Computer Vision and Pattern Recogni-tion Workshops (CVPRW ). Seattle, WA: IEEE, 2020: 2658-2664.
[18]	GARCIA-AUNON P, ROLD ′AN J J, BARRIENTOS
	A. Monitoring tra?c in future cities with aerial swarms: Developing and optimizing a behavior-based surveillance algorithm [J]. Cognitive Systems Research, 2019, 54: 273-286.
[19]	WAHYONO, JO K H. Cumulative dual foreground di?erences for illegally parked vehicles detection [J]. IEEE Transactions on Industrial Informatics, 2017, 13(5): 2464-2473.
[20]	XIE X M, WANG C Y, CHEN S, et al. Real-time il-legal parking detection system based on deep learning [C]//2017 International Conference on Deep Learning Technologies–ICDLT ’17. New York: ACM, 2017: 23-27.
[21]	CAI Y F, WANG H, CHEN X B, et al. Trajectory-based anomalous behaviour detection for intelligent tra?c surveillance [J]. IET Intelligent Transport Sys-tems, 2015, 9(8): 810-816.
[22]	WANG C, MUSAEV A, SHEINIDASHTEGOL P, et al. Towards detection of abnormal vehicle behavior us-ing tra?c cameras [C]//2019 International Conference on Big Data. Cham: Springer, 2019: 125-136.
[23]	WANG G A, YUAN X Y, ZHENG A T, et al. Anomaly candidate identi?cation and starting time estimation of vehicles from tra?c videos [C]//AI City Challenge Workshop IEEE/CVF Computer Vision and Pattern Recognition (CVPR) Conference.LongBeach, CA: IEEE, 2019: 382-390.