Hypergraph-Based Asynchronous Event Processing for Moving Object Classification

doi:10.1007/s12204-024-2699-y

Abstract

Abstract: Unlike traditional video cameras, event cameras capture asynchronous event streams in which each event encodes pixel location, triggers’ timestamps, and the polarity of brightness changes. In this paper, we introduce a novel hypergraph-based framework for moving object classification. Specifically, we capture moving objects with an event camera, to perceive and collect asynchronous event streams in a high temporal resolution. Unlike stacked event frames, we encode asynchronous event data into a hypergraph, fully mining the high-order correlation of event data, and designing a mixed convolutional hypergraph neural network for training to achieve a more efficient and accurate motion target recognition. The experimental results show that our method has a good performance in moving object classification (e.g., gait identification).

Key words: hypergraph learning, event stream, moving object classification

摘要： 与传统相机的采样方式不同，事件相机可以捕捉异步的事件流数据，其中每个事件由编码像素位置、触发事件和亮度的极性变化表示。本文针对传统相机在处理高动态复杂场景时的局限性问题，提出了一种基于异步事件流超图网络的运动目标步态识别新方法。具体来说, 本文先使用事件相机感知运动目标，采集高时间、空间分辨率的异步事件流数据。与堆叠事件帧的处理方法不同，本文结合超图网络模型进行学习，将异步事件数据编码成超图结构，充分挖掘事件数据的高阶相关性，并设计混合卷积的超图神经网络进行训练，在超图注意力机制和残差结构的优化下实现更高效、更准确的运动目标识别。实验结果表明，所提出的超图结构在处理高动态数据时性能更为优越，在公开的运动目标识别（如步态识别）数据集上具有前沿的效果。

关键词: 超图学习，事件流，运动目标识别

CLC Number:

TP181

YU Nannan, WANG Chaoyi, QIAO Yu, WANG Yuxin, ZHENG Chenglin, ZHANG Qiang, YANG Xin. Hypergraph-Based Asynchronous Event Processing for Moving Object Classification[J]. J Shanghai Jiaotong Univ Sci, 2025, 30(5): 952-961.

References

[1] LI J, TIAN Y. RECent advances in neuromorphic vision sensors: A survey [J]. Chinese Journal of Computers: 2021, 44(6): 1258-1286 (in Chinese).

[2] ERNST M O, BANKS M S. Humans integrate visual and haptic information in a statistically optimal fashion [J]. Nature, 2002, 415(6870): 429-433.

[3] HAN J, BHANU B. Individual recognition using gait energy image [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(2): 316-322.

[4] LEE L, GRIMSON W E L. Gait analysis for recognition and classification [C]//Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition. Washington: IEEE, 2002: 155-162.

[5] WANG L, TAN T N, NING H Z, et al. Silhouette analysis-based gait recognition for human identification [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(12): 1505-1518.

[6] TAO D C, LI X L, WU X D, et al. General tensor discriminant analysis and Gabor features for gait recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(10): 1700-1715.

[7] WU Z F, HUANG Y Z, WANG L, et al. A comprehensive study on cross-view gait based human identification with deep CNNs [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(2): 209-226.

[8] SHIRAGA K, MAKIHARA Y, MURAMATSU D, et al. GEINet: View-invariant gait recognition using a convolutional neural network [C]//2016 International Conference on Biometrics. Halmstad: IEEE, 2016: 1-8.

[9] WOLF T, BABAEE M, RIGOLL G. Multi-view gait recognition using 3D convolutional neural networks [C]//2016 IEEE International Conference on Image Processing. Phoenix: IEEE, 2016: 4165-4169.

[10] ALOTAIBI M, MAHMOOD A. Improved gait recognition based on specialized deep convolutional neural network [J]. Computer Vision and Image Understanding, 2017, 164: 103-110.

[11] LIU Z Y, SARKAR S. Simplest representation yet for gait recognition: Averaged silhouette [C]//Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR. Cambridge: IEEE, 2004: 211-214.

[12] NILSBACK M E, ZISSERMAN A. Automated flower classification over a large number of classes [C]//2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. Bhubaneswar: IEEE, 2008: 722-729.

[13] AMIR A, TABA B, BERG D, et al. A low power, fully event-based gesture recognition system [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 7388-7397.

[14] LAGORCE X, ORCHARD G, GALLUPPI F, et al. HOTS: A hierarchy of event-based time-surfaces for pattern recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(7): 1346-1359.

[15] PARK P K J, LEE K, LEE J H, et al. Computationally efficient, real-time motion recognition based on bio-inspired visual and cognitive processing [C]//2015 IEEE International Conference on Image Processing. Quebec City: IEEE, 2015: 932-935.

[16] BI Y, CHADHA A, ABBAS A, et al. Graph-based spatio-temporal feature learning for neuromorphic vision sensing [J]. IEEE Transactions on Image Processing, 2020, 29: 9084-9098.

[17] GAO G, KYRARINI M, RAZAVI M, et al. Comparison of Dynamic Vision Sensor-Based and IMU-based systems for ankle joint angle gait analysis [C]//2016 2nd International Conference on Frontiers of Signal Processing. Warsaw: IEEE, 2016: 93-98.

[18] LI Y J, ZHOU H, YANG B B, et al. Graph-based asynchronous event processing for rapid object recognition [C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 914-923.

[19] SCHAEFER S, GEHRIG D, SCARAMUZZA D. AEGNN: asynchronous event-based graph neural networks [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 12361-12371.

[20] WANG Y X, ZHANG X, SHEN Y R, et al. Event-stream representation for human gaits identification using deep neural networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7): 3436-3449.

[21] BERGE C. Hypergraphs: combinatorics of finite sets[M]. Amsterdam: Elsevier, 1984.

[22] KAHNG A B. Fast hypergraph partition [C]//Proceedings of the 26th ACM/IEEE Design Automation Conference. Las Vegas: ACM, 1989: 762-766.

[23] ALON N. Transversal numbers of uniform hypergraphs [J]. Graphs and Combinatorics, 1990, 6(1): 1-4.

[24] GUNOPULOS D, MANNILA H, KHARDON R, et al. Data mining, hypergraph transversals, and machine learning (extended abstract) [C]//Sixteenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems. Tucson: ACM, 1997: 209-216.

[25] CHUNG F. The Laplacian of a hypergraph [M]//Expanding graphs. Providence: AMS, 1992: 21-36.

[26] EITER T, GOTTLOB G. Identifying the minimal transversals of a hypergraph and related problems [J]. SIAM Journal on Computing, 1995, 24(6): 1278-1304.

[27] KARYPIS G, AGGARWAL R, KUMAR V, et al. Multilevel hypergraph partitioning: Applications in VLSI domain [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 1999, 7(1): 69-79.

[28] BULÒ S R, PELILLO M. A game-theoretic approach to hypergraph clustering [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 35: 1312-1327.

[29] PAPA D, MARKOV I. Hypergraph partitioning and clustering [M]// Handbook of approximation algorithms and metaheuristics. Boca Raton: Chapman and Hall/CRC, 2007: 61.

[30] SUN L, JI S W, YE J P. Hypergraph spectral learning for multi-label classification [C]// 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Las Vegas: ACM, 2008: 668-676.

[31] ZASS R, SHASHUA A. Probabilistic graph and hypergraph matching [C]//2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage: IEEE, 2008: 1-8.

[32] BRETTO A, GILLIBERT L. Hypergraph-based image representation [M]//Graph-based representations in pattern recognition. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005: 1-11.

[33] GUI H, LIU J L, TAO F B, et al. Large-scale embedding learning in heterogeneous event data [C]//2016 IEEE 16th International Conference on Data Mining. Barcelona: IEEE, 2016: 907-912.

[34] TU K, CUI P, WANG X A, et al. Structural deep embedding for hyper-networks [J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 426-433.

[35] ZHANG R C, ZOU Y S, MA J. Hyper-SAGNN: A self-attention based graph neural network for hypergraphs [DB/OL]. (2019-11-06). https://arxiv.org/abs/1911.0261

[36] YADATI N, NIMISHAKAVI M, YADAV P, et al. HyperGCN: Hypergraph convolutional networks for semi-supervised learning and combinatorial optimisation [DB/OL]. (2018-09-07). https://arxiv.org/abs/1809.02589

[37] STOFFREGEN T, GALLEGO G, DRUMMOND T, et al. Event-based motion segmentation by motion compensation [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 7243-7252.

[38] ALONSO I, MURILLO A C. EV-SegNet: Semantic segmentation for event-based cameras [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Long Beach: IEEE, 2019: 1624-1633.

[39] MITROKHIN A, HUA Z Y, FERMÜLLER C, et al. Learning visual motion segmentation using event surfaces [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 14402-14411.

[40] BAI S, ZHANG F H, TORR P H S. Hypergraph convolution and hypergraph attention [J]. Pattern Recognition, 2021, 110: 107637.

[41] AGARWAL S, BRANSON K, BELONGIE S. Higher order learning with graphs [C]// 23rd International Conference on Machine Learning. Pittsburgh: ACM, 2006: 17-24.

[42] BI Y, CHADHA A, ABBAS A, et al. Graph-based object classification for neuromorphic vision sensing [C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 491-501.