图卷积网络融合脑电与肌电用于上肢运动意图识别

doi:10.1007/s12204-025-2856-y

摘要/Abstract

摘要： 随着传感器和算法的不断进步，越来越多的深度学习方法被应用于利用多模态生理信号进行精细化的上肢运动意图识别。然而，如何有效且可量化地整合脑电（EEG）与肌电（EMG）信号通道之间，以及脑电信号通道内部的相关性，仍然是提升识别性能的一大挑战。本文提出了一种神经网络结构，通过融合脑电和肌电信号，实现对上肢运动意图的准确预测。首先，将原始输入信号输入至特征提取模块，实现通道维度上的特征分解。其次，提出了一种边权可学习的图卷积模块，用于自适应学习不同模态之间的相关性。第三，设计了一种自注意力图池化模块，通过自注意力机制计算每个节点的注意力分数，作为池化的依据；相比采用均值或最大值的计算方法，该方法能更有效地保留与运动意图相关性更强的节点。最后，通过分类器得到最终预测结果。我们在一个公开的多模态上肢数据集上验证了该方法的有效性，准确率达到 93.17%。

关键词: 运动意图识别, 图卷积网络, 脑电图, 肌电图, 多模态

Abstract: With the continuous advancement of sensors and algorithms, an increasing number of deep learning methods have been applied to fine-grained upper limb motion intention recognition using multimodal physiological signals. However, effectively and quantifiably integrating correlations between electroencephalogram (EEG) and electromyogram (EMG) signal channels as well as within EEG signal channels as a clue to improve performance remained challenging. In this paper, we proposed a novel framework that achieved accurate prediction of upper limb motion intentions via fusing EEG and EMG signals. Firstly, the raw input signals were fed into the feature extraction module, respectively, enabling feature decomposition in the channel dimension. Secondly, the graph convolution module with learnable edge weights was proposed to adaptively learn correlations between different modalities. Thirdly, we designed a self-attention graph pooling module that employed the self-attention mechanism to compute the attention score for each node as the basis for pooling. Compared with calculation methods using the mean or maximum value, this approach was more likely to retain nodes with stronger correlations to motor intentions. Finally, the prediction results were obtained through a classifier. We validated the effectiveness of our method on a publicly available multimodal upper limb dataset, achieving an accuracy of 93.17%.

中图分类号:

TP391.4
R318

. 图卷积网络融合脑电与肌电用于上肢运动意图识别[J]. J Shanghai Jiaotong Univ Sci, 2026, 31(1): 12-23.

Zheng Luzhou, Zhao Changchen, Zhang Chao, Cheng Shichao, Zhang Jianhai. Graph Convolution Network with EEG-EMG Fusion for Upper Limb Motion Intention Recognition[J]. J Shanghai Jiaotong Univ Sci, 2026, 31(1): 12-23.

参考文献

[1] VOLPE B T, KREBS H I, HOGAN N, et al. A novel approach to stroke rehabilitation: Robot-aided sensorimotor stimulation [J]. Neurology, 2000, 54(10): 1938-1944.

[2] HE G S, HUANG X G, LI F, et al. Review of power-assisted lower limb exoskeleton robot [J]. Journal of Shanghai Jiao Tong University (Science), 2024, 29(1): 1-15.

[3] BANSIL S, PRAKASH N, KAYE J, et al. Movement disorders after stroke in adults: A review [J]. Tremor and Other Hyperkinetic Movements, 2012, 2: tre-02-42-195-1.

[4] TAO J, ZHOU Z H. Review of key technologies for developing personalized lower limb rehabilitative exoskeleton robots [J]. Journal of Shanghai Jiao Tong University (Science), 2024, 29(1): 16-28.

[5] MASENGO G, ZHANG X D, DONG R L, et al. Lower limb exoskeleton robot and its cooperative control: A review, trends, and challenges for future research [J]. Frontiers in Neurorobotics, 2022, 16: 913748.

[6] SU D N, HU Z G, WU J P, et al. Review of adaptive control for stroke lower limb exoskeleton rehabilitation robot based on motion intention recognition [J]. Frontiers in Neurorobotics, 2023, 17: 1186175.

[7] LI K X, ZHANG J H, LIU X, et al. Estimation of continuous elbow joint movement based on human physiological structure [J]. Biomedical Engineering Online, 2019, 18(1): 31.

[8] SARTORI M, LLYOD D G, FARINA D. Neural data-driven musculoskeletal modeling for personalized neurorehabilitation technologies [J]. IEEE Transactions on Bio-Medical Engineering, 2016, 63(5): 879-893.

[9] MOHD KHAIRUDDIN I, SIDEK S N, P P ABDUL MAJEED A, et al. The classification of movement intention through machine learning models: The identification of significant time-domain EMG features [J]. PeerJ Computer Science, 2021, 7: e379.

[10] ZHANG L, LIU G, HAN B, et al. sEMG based human motion intention recognition [J]. Journal of Robotics, 2019, 2019(1): 3679174.

[11] BUERKLE A, EATON W, LOHSE N, et al. EEG based arm movement intention recognition towards enhanced safety in symbiotic Human-Robot Collaboration [J]. Robotics and Computer-Integrated Manufacturing, 2021, 70: 102137.

[12] TRYON J, FRIEDMAN E, TREJOS A L. Performance evaluation of EEG/EMG fusion methods for motion classification [C]//2019 IEEE 16th International Conference on Rehabilitation Robotics. Toronto: IEEE, 2019: 971-976.

[13] LI Z J, YUAN Y X, LUO L, et al. Hybrid brain/muscle signals powered wearable walking exoskeleton enhancing motor ability in climbing stairs activity [J]. IEEE Transactions on Medical Robotics and Bionics, 2019, 1(4): 218-227.

[14] LI M Y, DUAN S C, DONG Y, et al. A hierarchical fusion strategy based on EEG and sEMG for human-exoskeleton system [C]//2020 IEEE International Conference on Real-time Computing and Robotics. Asahikawa: IEEE, 2020: 458-463.

[15] BIRD J J, KOBYLARZ J, FARIA D R, et al. Cross-domain MLP and CNN transfer learning for biological signal processing: EEG and EMG [J]. IEEE Access, 2020, 8: 54789-54801.

[16] LEW K L, SIM K S, TAN S C, et al. Biofeedback upper limb assessment using electroencephalogram, electromyographic and electrocardiographic with machine learning in signal classification [J]. Engineering Letters, 2022, 30(3): 935-947.

[17] SHI K C, MU F J, HUANG R, et al. Multimodal human-exoskeleton interface for lower limb movement prediction through a dense co-attention symmetric mechanism [J]. Frontiers in Neuroscience, 2022, 16: 796290.

[18] YANG S Q, LI M, WANG J L. Fusing sEMG and EEG to increase the robustness of hand motion recognition using functional connectivity and GCN [J]. IEEE Sensors Journal, 2022, 22(24): 24309-24319.

[19] LI H Y, JI H F, YU J, et al. A sequential learning model with GNN for EEG-EMG-based stroke rehabilitation BCI [J]. Frontiers in Neuroscience, 2023, 17: 1125230.

[20] ZHANG Y X, QIAO L K, ZHAO M R. Fault diagnosis for wind turbine generators using normal behavior model based on multi-task learning [J]. IEEE Transactions on Automation Science and Engineering, 2024, 21(2): 1258-1270.

[21] ZHAO C C, LIU K, ZHENG H, et al. Cross-modality self-attention and fusion-based neural network for lower limb locomotion mode recognition [J]. IEEE Transactions on Automation Science and Engineering, 2024, 22: 5411-5424.

[22] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// 31st Conference on Neural Information Processing Systems. Long Beach: NIPS, 2017: 1-11.

[23] ZHOU H Y, YANG G, WANG B C, et al. An attention-based deep learning approach for inertial motion recognition and estimation in human-robot collaboration [J]. Journal of Manufacturing Systems, 2023, 67: 97-110.

[24] DUTTA A K, RAPARTHI M, ALSAADI M, et al. Deep learning-based multi-head self-attention model for human epilepsy identification from EEG signal for biomedical traits [J]. Multimedia Tools and Applications, 2024, 83(33): 80201-80223.

[25] TAO W, LI C, SONG R C, et al. EEG-based emotion recognition via channel-wise attention and self attention [J]. IEEE Transactions on Affective Computing, 2023, 14(1): 382-393.

[26] JEONG J H, CHO J H, SHIM K H, et al. Multimodal signal dataset for 11 intuitive movement tasks from single upper extremity during multiple recording sessions [J]. GigaScience, 2020, 9(10): giaa098.

[27] SAWANGJAI P, TRAKULRUANGROJ M, BOONNAG C, et al. EEGANet: Removal of ocular artifacts from the EEG signal using generative adversarial networks [J]. IEEE Journal of Biomedical and Health Informatics, 2022, 26(10): 4913-4924.

[28] ZHANG W M, ZHANG X D, XU C, et al. An approach for upper limb movement intention recognition using EEG and sEMG fusion based on the MCPSA-CIIM [C]//2023 WRC Symposium on Advanced Robotics and Automation. Beijing: IEEE, 2023: 402-407.

[29] ZHANG R, CHEN Y D, XU Z X, et al. Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network [J]. Frontiers in Neuroscience, 2023, 17: 1129049.

[30] PARK S H, LEE D, LEE S G. Filter bank regularized common spatial pattern ensemble for small sample motor imagery classification [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(2): 498-505.

[31] KLEIN BRETELER M D, SIMURA K J, FLANDERS M. Timing of muscle activation in a hand movement sequence [J]. Cerebral Cortex, 2007, 17(4): 803-815.

[32] SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization [J]. Human Brain Mapping, 2017, 38(11): 5391-5420.

[33] LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces [J]. Journal of Neural Engineering, 2018, 15(5): 056013.

[34] AL-QURAISHI M S, ELAMVAZUTHI I, TANG T B, et al. Multimodal fusion approach based on EEG and EMG signals for lower limb movement recognition [J]. IEEE Sensors Journal, 2021, 21(24): 27640-27650.