Integrated Framework for Test and Evaluation of Autonomous Vehicles

doi:10.1007/s12204-021-2360-y

Abstract

Abstract: Autonomous vehicles must pass e?ective standard tests to verify their reliability and safety. Accord-ingly, it is very important to establish a complete scienti?c test and evaluation system for autonomous vehicles. A comprehensive framework incorporating the design of test scenarios, selection of evaluation indexes, and estab-lishment of an evaluation system is proposed in this paper. The aims of the system are to obtain an objective and quantitative score regarding the intelligence of autonomous vehicles, and to form an automated process in the future development. The proposed framework is built on a simulation platform to ensure the feasibility of the design and implementation of the test scenarios. The design principle for the test scenarios is also presented. To reduce subjective in?uences, the proposed framework selects objective indexes from four aspects: safety, comfort, driving performance, and standard regulations. The order relation analysis method is adopted to formulate the index weights, and fuzzy comprehensive evaluation is used to quantify the scores. Finally, a numerical example is provided to visually demonstrate the evaluation results for the autonomous vehicles scored by the proposed framework.

Key words: test and evaluation, scenario construction, evaluation index, autonomous vehicles

CLC Number:

U 467

SU Yimin (苏奕敏), WANG Lin∗ (王琳). Integrated Framework for Test and Evaluation of Autonomous Vehicles[J]. J Shanghai Jiaotong Univ Sci, 2021, 26(5): 699-712.

References 32

[1]	HUANG H, ZHENG X, YANG Y, et al. An integrated architecture for intelligence evaluation of automated vehicles [J]. Accident Analysis & Prevention, 2020, 145: 105681.
[2]	HUANG H M, PAVEK K, NOVAK B, et al. A frame-work for autonomy levels for unmanned systems (AL-FUS) [C]//AUVSI’s Unmanned Systems North Amer-ica. Baltimore: NIST, 2005: 1-9.
[3]	DARPA. Urban challenge rules [DB/OL]. (2007-10-27)[2020-11-20]. https://www.grandchallenge.org/ grand-challenge/docs/Urban Challenge Rules 102707.pdf.
[4]	SUN Y, CHEN H. Research on test and evaluation of unmanned ground vehicles [J]. Acta Armamentarii, 2015, 36(6): 978-986 (in Chinese).
[5]	SUN Y, XIONG G, CHEN H. Evaluation of the in-telligent behaviors of unmanned ground vehicles based on fuzzy-EAHP scheme [J]. Automotive Engineering, 2014, 36(1): 22-27 (in Chinese).
[6]	SUN Y. Quantitative evaluation of intelligence levels for unmanned ground vehicles [D]. Beijing: Beijing In-stitute of Technology, 2014 (in Chinese).
[7]	XIONG G M, GAO L, WU S B, et al. Intelligent be-haviors and test and evaluation for unmanned ground vehicles [M]. Beijing: Beijing Institute of Technology Press, 2015 (in Chinese).
[8]	SON T D, BHAVE A, VAN DER AUWERAER H. Simulation-based testing framework for autonomous driving development [C]//2019 IEEE International Conference on Mechatronics(ICM ). Ilmenau: IEEE, 2019: 576-583.
[9]	WANG G, DENG W, ZHANG S, et al. A comprehen-sive testing and evaluation approach for autonomous vehicles [J]. SAE Technical Paper, 2018: 2018-01-0124.
[10]	WENG B, RAO S J, DEOSTHALE E, et al. Model predictive instantaneous safety metric for evaluation of automated driving systems [C]//IEEE Intelligent Ve-hicles Symposium (IV ). Las Vegas: IEEE, 2020: 1899-1906.
[11]	LI L, HUANG W, LIU Y, et al. Intelligence testing for autonomous vehicles: A new approach [J]. IEEE Transactions on Intelligent Vehicles, 2016, 1(2): 158-166.
[12]	FENG S, FENG Y, YU C, et al. Testing scenario li-brary generation for connected and automated vehi-cles, part I: Methodology [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3): 1573-1582. [13] FENG S, FENG Y, SUN H, et al. Testing scenario library generation for connected and automated vehi-cles, part II: case studies [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(9): 5635-5647.
[14]	SAE International. Taxonomy and de?nitions for terms related to driving automation systems for on-road motor vehicles: J3016 [R]. Warrendale: SAE, 2018.
[15]	YU Z, XING X, CHEN J. Review on automated vehi-cle testing technology and its application [J]. Journal of Tongji University (Natural Science), 2019, 47(4):
54	0- 547 (in Chinese).
[16]	PEGASUS. PEGASUS joint project [DB/OL].(2019-05-14) [2020-11-20]. http://www.pegasuspro-jekt.de/en/.
[17]	MENZEL T, BAGSCHIK G, MAURER M. Scenar-ios for development, test and validation of automated vehicles [C]//2018 IEEE Intelligent Vehicles Sympo-sium(IV ). Changshu: IEEE, 2018: 1821-1827.
[18]	WATANABE H, TOBISCH L, ROST J, et al. Sce-nario mining for development of predictive safety func-tions [C]//2019 IEEE International Conference on Ve-hicular Electronics and Safety(ICVES ). Cairo: IEEE, 2019: 1-7.
[19]	HighD Dataset. HighD dataset [DB/OL]. [2020-11-20]. https: //www.highd-dataset.com/.
[20]	DATA.GOV. Safety pilot model deployment data [DB/OL]. (2020-08-21) [2020-11-20]. https://catalog. data.gov/dataset/safety-pilot-model-deployment-data.
[21]	UMTRI. Safety pilot model deployment. [DB/OL].[2020-11-20]. http://safetypilot.umtri.umich.edu/.
[22]	RAHMAN M S, ABDEL-ATY M, LEE J, et al. Safety bene?ts of arterials’ crash risk under connected and automated vehicles [J]. Transportation Research Part C: Emerging Technologies, 2019, 100: 354-371. [23] CHEN J, LI R, XING X, et al. Survey on intelli-gence evaluation of autonomous vehicles [J]. Journal of Tongji University (Natural Science), 2019, 47(12): 1785-1790 (in Chinese).
[24]	ZHAO Y N, MENG K W, GAO L. The entropy-cost function evaluation method for unmanned ground vehicles [J]. Mathematical Problems in Engineering, 2015, 2015:1-6.
[25]	HAYWARD J C. Near-miss determination through use of a scale of danger [C]//51st Annual Meeting of the Highway Research Board. Washington, DC: Highway Research Board, 1972: 24-34.
[26]	Standardization Administration. Intelligent trans-portation systems: Forward vehicle collision warning systems: Performance requirements and test proce-dures: GB/T 33577—2017 [S]. Beijing: Standards Press of China, 2017 (in Chinese).
[27]	MINDERHOUD M M, BOVY P H L. Extended time-to-collision measures for road tra?c safety assessment [J]. Accident Analysis & Prevention, 2001, 33(1): 89-97.
[28]	BAGDADI O, VA′ RHELYI A. Jerky driving: An indi-cator of accident proneness? [J]. Accident Analysis & Prevention, 2011, 43(4): 1359-1363.
[29]	BELLEM H, SCHOENENBERG T, KREMS J F, et al. Objective metrics of comfort: developing a driving style for highly automated vehicles [J]. Transportation Research Part F: Tra?c Psychology and Behaviour, 2016, 41: 45-54.
[30]	GUO Y. Comprehensive evaluation theory and method [M]. Beijing: Science Press, 2002 (in Chinese).
[31]	XIA Q, LIU K, HUANG L, et al. Study of compre-hensive evaluation for L2 automated vehicles on ?eld test [C]//2019 3rd Conference on Vehicle Control and Intelligence (CVCI ). Hefei: IEEE, 2019: 1-6.
[32]	WANG X H, LI T J, DING L L. Evaluation theory and technology of complex large-scale systems [M]. Jinan: Shandong University Press, 2010 (in Chinese).