Vehicle-Road Collaborative Perception Method Based on Dual-Stream Feature Extraction

doi:10.16183/j.cnki.jsjtu.2024.239

Abstract

Abstract:

To solve the problem of inadequate perception of autonomous driving in occlusion and over-the-horizon scenarios, a vehicle-road collaborative perception method based on a dual-stream feature extraction network is proposed to enhance the 3D object detection capabilities of traffic participants. Feature extraction networks for roadside and vehicle-side scenes are tailored based on respective characteristics. Since roadside has rich and sufficient sensing data and computational resources, the Transformer structure is used to extract more sophisticated and advanced feature representations. Due to limited computational capability and high real-time demands of autonomous vehicles, partial convolution (PConv) is employed to enhance computing efficiency, and the Mamba-VSS module is introduced for efficient perception in complex environments. Collaborative perception between vehicle-side and roadside is accomplished through the selective sharing and fusion of critical perceptual information guided by confidence maps. By training and testing on DAIR-V2X dataset, the model size of vehicle-side feature extraction network is obtained to be 8.1 MB, and the IoU thresholds of 0.5 and 0.7 correspond to the average accuracy indexes of 67.67% and 53.74%. The experiment verifies the advantages of this method in detection accuracy and model size, and provides a lower-configuration detection scheme for vehicle-road collaboration.

Key words: autonomous driving, collaborative perception, feature extraction, 3D object detection, information sharing and fusion

CLC Number:

TP391.4

NIU Guochen, SUN Xiangyu, YUAN Zhengyan. Vehicle-Road Collaborative Perception Method Based on Dual-Stream Feature Extraction[J]. Journal of Shanghai Jiao Tong University, 2024, 58(11): 1826-1834.

Figures/Tables 9

Fig.1

Fig.2

Fig.3

Tab.1

Tab.2

Fig.4

Tab.3

Fig.5

Fig.6

References 21

[1]	伊笑莹, 芮一康, 冉斌, 等. 车路协同感知技术研究进展及展望[J]. 中国工程科学, 2024, 26(1): 178-189. doi: 10.15302/J-SSCAE-2024.01.016
	YI Xiaoying, RUI Yikang, RAN Bin, et al. Vehicle-infrastructure cooperative sensing: Progress and prospect[J]. Strategic Study of CAE, 2024, 26(1): 178-189. doi: 10.15302/J-SSCAE-2024.01.016
[2]	ARNOLD E, DIANATI M, DE TEMPLE R, et al. Cooperative perception for 3D object detection in driving scenarios using infrastructure sensors[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(3): 1852-1864.
[3]	张毅, 姚丹亚, 李力, 等. 智能车路协同系统关键技术与应用[J]. 交通运输系统工程与信息, 2021, 21(5): 40-51.
	ZHANG Yi, YAO Danya, LI Li, et al. Technologies and applications for intelligent vehicle-infrastructure cooperation systems[J]. Journal of Transportation Systems Engineering and Information Technology, 2021, 21(5): 40-51.
[4]	DOSOVITSKIY A, ROS G, CODERVILLA F, et al. CARLA: An open urban driving simulator[C]//1st Conference on Robot Learning. Mountain View, USA: CoRL, 2017: 5550767.
[5]	CHEN Q, TANG S H, YANG Q, et al. Cooper: Cooperative perception for connected autonomous vehicles based on 3D point clouds[C]//2019 IEEE 39th International Conference on Distributed Computing Systems. Dallas, TX, USA: IEEE, 2019: 514-524.
[6]	CHEN Q. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds[DB/OL]. (2019-09-13)[2024-06-10]. https://arxiv.org/abs/1909.06459.
[7]	GUO J D, CARRILLO D, TANG S H, et al. CoFF: Cooperative spatial feature fusion for 3-D object detection on autonomous vehicles[J]. IEEE Internet of Things Journal, 2021, 8(14): 11078-11087.
[8]	HU Y, FANG S, LEI Z, et al.Where2comm: Communication-efficient collaborative perception via spatial confidence maps[C]//36th Corference on Neural Information Processing Systems. New Orleans, USA: NIPS, 2022: 4874-4886.
[9]	LIU Y C, TIAN J J, GLASER N, et al. When2com: Multi-agent perception via communication graph grouping[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020: 4105-4114.
[10]	WANG J Y, ZENG Y, GONG Y. Collaborative 3D object detection for automatic vehicle systems via learnable communications[DB/OL]. (2022-05-24) [2024-06-10]. https://arxiv.org/abs/2205.11849v1.
[11]	王秉路, 靳杨, 张磊, 等. 基于多传感器融合的协同感知方法[J]. 雷达学报, 2024, 13(1): 87-96.
	WANG Binglu, JIN Yang, ZHANG Lei, et al. Collaborative perception method based on multisensor fusion[J]. Journal of Radars, 2024, 13(1): 87-96.
[12]	LANG A H, VORA S, CAESAR H, et al. PointPillars: Fast encoders for object detection from point clouds[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 12689-12697.
[13]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[DB/OL]. (2020-10-22)[2024-06-10]. https://arxiv.org/abs/2010.11929.
[14]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal, QC, Canada: IEEE, 2021: 9992-10002.
[15]	CHEN J R, KAO S H, HE H, et al. Run, don’t walk: Chasing higher FLOPS for faster neural networks[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, BC, Canada: IEEE, 2023: 12021-12031.
[16]	GU A, DAO T. Mamba: Linear-time sequence modeling with selective state spaces[DB/OL]. (2023-12-01)[2024-06-10]. https://arxiv.org/abs/2312.00752v2.
[17]	上海交通大学. 基于空间置信度图的多轮多模态多智能体的协同感知方法: CN 202211076556.X[P]. 2022-12-13[2024-06-10].
[18]	YU H B, LUO Y Z, SHU M, et al. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3D object detection[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, 2022: 21329-21338.
[19]	WANG T H, MANIVASAGAM S, LIANG M, et al. V2VNet: Vehicle-to-Vehicle communication for joint perception and prediction[M]//Computer Vision-ECCV 2020. Cham: Springer, 2020: 605-621.
[20]	XU R S, XIANG H, TU Z Z, et al. V2X-ViT: Vehicle-to-Everything cooperative perception with Vision Transformer[M]//Computer Vision-ECCV 2022. Cham: Springer, 2022: 107-124.
[21]	MEHR E, JOURDAN A, THOME N, et al. DiscoNet: Shapes learning on disconnected manifolds for 3D editing[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3473-3482.

网络	方法	AP30/%	AP50/%	AP70/%	模型大小/MB		GFLOPs
轻量化单主干网络	MobileNetV2	59.54	53.19	34.25	3.5		0.3
	FasterNet	63.10	57.25	38.75	7.6		0.85
	YOLO-backbone	62.19	55.70	36.11	5.1		2.1
	EfficientFormerv2	60.76	55.85	38.60	6.1		2.7
大规模单主干网络	ResNet	66.80	62.14	46.50	61.0		10.1
	ConvNextv2	66.57	61.61	46.60	89.0		15.4
	Swin-Transformer	71.17	66.36	52.44	88.0		15.4
车路双流主干网络	本文方法	72.35	67.67	53.74	车端,8.1	路端,88	车端,1.1	路端,15.8

Baseline	PConv	Mamba- VSS	AP30/ %	AP50/ %	AP70/ %	模型大小/MB
√			67.54	61.27	51.82	13.5
√	√		71.11	64.48	51.05	9.7
√	√	√	72.35	67.67	53.74	8.1

方法	AP50/%	AP70/%	C
No Collaboration	50.03	43.57	0.00
Late Fusion	53.12	37.88	11.45
When2com	51.12	36.17	22.62
V2VNet	56.01	42.25	24.21
V2X-ViT	54.26	43.35	22.62
DiscoNet	54.29	44.88	22.62
Where2comm	63.54	48.78	20.56
本文方法	67.67	53.74	20.56