综述：运动想像脑机接口中的迁移学习

doi:10.1007/s12204-022-2488-4

摘要/Abstract

摘要： 迁移学习是一种新的机器学习方法，能够利用现有知识解决相关但不相同域的问题。在训练数据不足的情况下，它常将另一个域的训练数据进行迁移用于模型训练。近年来，越来越多的脑机接口研究者关注迁移学习，充分利用从不同受试者获取的脑电数据，有效降低数据采集和标签的成本，同时极大地改善模型学习性能。本文对迁移学习的发展进行调查，并综述了脑机接口中的迁移学习方法。依据迁移学习中“迁移什么”问题，本文分为三部分：基于实例的迁移学习、基于参数的迁移学习和基于特征的迁移学习。另外，将目前脑机接口研究中的迁移学习应用情况从迁移学习方法、数据库和性能评价等方面进行概述。最后，提出未来研究要解决的问题，对脑机接口中迁移学习的普及与深入研究奠定了基础。

关键词: 迁移学习，脑机接口，脑电图，机器学习

Abstract: Transfer learning, as a new machine learning methodology, may solve problems in related but different domains by using existing knowledge, and it is often applied to transfer training data from another domain for model training in the case of insufficient training data. In recent years, an increasing number of researchers who engage in brain-computer interface (BCI), have focused on using transfer learning to make most of the available electroencephalogram data from different subjects, effectively reducing the cost of expensive data acquisition and labeling as well as greatly improving the learning performance of the model. This paper surveys the development of transfer learning and reviews the transfer learning approaches in BCI. In addition, according to the “what to transfer” question in transfer learning, this review is organized into three contexts: instance-based transfer learning, parameter-based transfer learning, and feature-based transfer learning. Furthermore, the current transfer learning applications in BCI research are summarized in terms of the transfer learning methods, datasets, evaluation performance, etc. At the end of the paper, the questions to be solved in future research are put forward, laying the foundation for the popularization and in-depth research of transfer learning in BCI.

Key words: transfer learning, brain-computer interface (BCI), electroencephalogram, machine learning

中图分类号:

TP181
R318

李明爱1,2,3，许冬芹1. 综述：运动想像脑机接口中的迁移学习[J]. J Shanghai Jiaotong Univ Sci, 2024, 29(1): 37-59.

LI Mingai1,2,3∗ (李明爱), XU Dongqin1 (许东芹). Transfer Learning in Motor Imagery Brain Computer Interface: A Review[J]. J Shanghai Jiaotong Univ Sci, 2024, 29(1): 37-59.

参考文献

[1] ZHUANG F Z, LUO P, HE Q, et al. Survey on transfer learning research [J]. Journal of Software, 2015, 26(1): 26-39 (in Chinese).
[2] NICOLAS-ALONSO L F, GOMEZ-GIL J. Brain computer interfaces, a review [J]. Sensors, 2012, 12(2): 1211-1279.
[3] LOTTE F, BOUGRAIN L, CICHOCKI A, et al. A review of classification algorithms for EEG-based braincomputer interfaces: A 10 year update [J]. Journal of Neural Engineering, 2018, 15(3): 031005.
[4] PAN S J, YANG Q. A survey on transfer learning [J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.
[5] WEISS K, KHOSHGOFTAAR T M, WANG D D. A survey of transfer learning [J]. Journal of Big Data, 2016, 3: 9.
[6] DAY O, KHOSHGOFTAAR T M. A survey on heterogeneous transfer learning [J]. Journal of Big Data, 2017, 4: 29.
[7] ARGYRIOU A, MAURER A, PONTIL M. An algorithm for transfer learning in a heterogeneous environment [M]//Machine learning and knowledge discovery in databases. Berlin, Heidelberg: Springer, 2008: 71-85.
[8] YIN J, YANG Q, NI L. Adaptive temporal radio maps for indoor location estimation [C]//Third IEEE International Conference on Pervasive Computing and Communications. Kauai, HI: IEEE, 2005: 85-94.
[9] ZHANG L N, ZUO X, LIU J W. Research and development on zero-shot learning [J]. Acta Automatica Sinica, 2020, 46(1): 1-23 (in Chinese).
[10] TAYLOR M E, STONE P. An introduction to intertask transfer for reinforcement learning [J]. AI Magazine, 2011, 32(1): 15.
[11] TAYLOR M E, STONE P. Transfer learning for reinforcement learning domains: A survey [J]. Journal of Machine Learning Research, 2009, 10(10): 1633-1685.
[12] SU L J, YAO M, ZHENG N G, et al. Correlation between extreme learning machine and entorhinal hippocampal system [C]//Proceedings of ELM-2015 Volume 2. Cham: Springer, 2016: 307-315.
[13] WU Z H, ZHENG N G, ZHANG S W, et al. Maze learning by a hybrid brain-computer system [J]. Scientific Reports, 2016, 6: 31746.
[14] KONG X Y, GONG S, SU L J, et al. Neuroprotective effects of allicin on ischemia-reperfusion brain injury [J]. Oncotarget, 2017, 8(61): 104492-104507.
[15] TAN C Q, SUN F C, KONG T, et al. Attentionbased transfer learning for brain-computer interface [C]//2019 IEEE International Conference on Acoustics, Speech and Signal Processing.
Brighton: IEEE, 2019: 1154-1158.
[16] PAVONE K J, SU L J, GAO L, et al. Lack of responsiveness during the onset and offset of sevoflurane anesthesia is associated with decreased awake-alpha oscillation power [J]. Frontiers in Systems Neuroscience, 2017, 11: 38.
[17] JAYARAM V, ALAMGIR M, ALTUN Y, et al. Transfer learning in brain-computer interfaces [J]. IEEE Computational Intelligence Magazine, 2016, 11(1): 20-31.
[18] GU X T, CAO Z H, JOLFAEI A, et al. EEG-based brain-computer interfaces (BCIs): A survey of recent studies on signal sensing technologies and computational intelligence approaches and their applications [J]. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021, 18(5): 1645-1666.
[19] HOSSAIN I, KHOSRAVI A, HETTIARACHCHI I, et al. Multiclass informative instance transfer learning framework for motor imagery-based brain-computer interface [J]. Computational Intelligence and Neuroscience, 2018, 2018: 6323414.
[20] WU H, NIU Y, LI F, et al. A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification [J]. Frontiers in Neuroscience, 2019, 13: 1275.
[21] AZAB A M, MIHAYLOVA L, ANG K K, et al. Weighted transfer learning for improving motor imagery-based brain-computer interface [J]. IEEETransactions on Neural Systems and Rehabilitation Engineering, 2019, 27(7): 1352-1359.
[22] ARVANEH M, GUAN C T, ANG K K, et al. EEG data space adaptation to reduce intersession nonstationarity in brain-computer interface [J]. Neural Computation, 2013, 25(8): 2146-2171.
[23] RODRIGUES P L C, CONGEDO M, JUTTEN C. Dimensionality transcending: A method for merging BCI datasets with different dimensionalities [J]. IEEE Transactions on Biomedical Engineering, 2021, 68(2): 673-684.
[24] ZHENG M M, YANG B H, XIE Y L. EEG classification across Sessions and across subjects through transfer learning in motor imagery-based brain-machine interface system [J]. Medical & Biological Engineering & Computing, 2020, 58(7): 1515-1528.
[25] HUANG Z H, ZHENG W M, WU Y J, et al. Ensemble or pool: A comprehensive study on transfer learning for c-VEP BCI during interpersonal interaction [J]. Journal of Neuroscience Methods, 2020, 343: 108855.
[26] SAHA S, BAUMERT M. Intra- and inter-subject variability in EEG-based sensorimotor brain computer interface: A review [J]. Frontiers in Computational Neuroscience, 2020, 13: 87.
[27] XU L C, XU M P, KE Y F, et al. Cross-dataset variability problem in EEG decoding with deep learning [J]. Frontiers in Human Neuroscience, 2020, 14: 103.
[28] ROY S, CHOWDHURY A, MCCREADIE K, et al. Deep learning based inter-subject continuous decoding of motor imagery for practical brain-computer interfaces [J]. Frontiers in Neuroscience, 2020, 14: 918.
[29] ZHANG K, XU G H, CHEN L, et al. Instance transfer subject-dependent strategy for motor imagery signal classification using deep convolutional neural networks [J]. Computational and Mathematical Methods in Medicine, 2020, 2020: 1683013.
[30] PARVAN M, GHIASI A R, REZAII T Y, et al. Transfer Learning based Motor Imagery Classification using Convolutional Neural Networks [C]//2019 27th Iranian Conference on Electrical Engineering (ICEE). Yazd: IEEE, 2019: 1825-1828.
[31] KSHIRSAGAR G B, LONDHE N D. Weighted ensemble of deep convolution neural networks for single-trial character detection in Devanagari-script-based P300 speller [J]. IEEE Transactions on Cognitive and Developmental Systems, 2020, 12(3): 551-560.
[32] KOSTAS D, RUDZICZ F. Thinker invariance: Enabling deep neural networks for BCI across more people [J]. Journal of Neural Engineering, 2020, 17(5): 056008.
[33] KANT P, LASKAR S H, HAZARIKA J, et al. CWT based transfer learning for motor imagery classification for brain computer interfaces [J]. Journal of Neuroscience Methods, 2020, 345: 108886.
[34] WANG Y, ZHANG M, WU R M, et al. Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities [J]. Brain Sciences, 2020, 10(7): 442.
[35] THRUN S, PRATT L. Learning to learn: Introduction and overview [M]//Learning to learn. Boston: Springer, 1998: 3-17.
[36] LI Y, KAMBARA H, KOIKE Y, et al. Application of covariate shift adaptation techniques in brain-computer interfaces [J]. IEEE Transactions on Biomedical Engineering, 2010, 57(6): 1318-1324.
[37] LI Y, KOIKE Y, SUGIYAMA M. A framework of adaptive brain computer interfaces [C]//2009 2nd International Conference on Biomedical Engineering and Informatics. Tianjin: IEEE, 2009: 1-5.
[38] WU D R, LANCE B J, PARSONS T D. Collaborative filtering for brain-computer interaction using transfer learning and active class selection [J]. PLoS ONE, 2013, 8(2): e56624.
[39] HOSSAIN I, KHOSRAVI A, NAHAVANDHI S. Active transfer learning and selective instance transfer with active learning for motor imagery based BCI [C]//2016 International Joint Conference on Neural Networks. Vancouver, BC: IEEE, 2016: 4048-4055.
[40] HOSSAIN I, KHOSRAVI A, HETTIARACHCHI I T, et al. Informative instance transfer learning with subject specific frequency responses for motor imagery brain computer interface [C]//2017 IEEE International Conference on Systems, Man, and Cybernetics. Banff, AB: IEEE, 2017: 252-257.
[41] WU D R. Active semi-supervised transfer learning (ASTL) for offline BCI calibration [C]//2017 IEEE International Conference on Systems, Man, and Cybernetics. Banff, AB: IEEE, 2017: 246-251.
[42] VERHOEVEN T, VUYLSTEKER B, DAMBRE J. Model selection for subject-to-subject transfer learning in brain-computer interfaces [EB/OL]. [2021-03-29]. https://www.researchgate.net/publication 318788804 Model Selection for Subject-to-Subject Transfer Learning in Brain-Computer Interfaces.
[43] WEI C S, LIN Y P, WANG Y T, et al. Transfer learning with large-scale data in brain-computer interfaces [C]//2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Orlando, FL: IEEE, 2016: 4666-4669.
[44] FAUZI H, SHAPIAI M I, KHAIRUDDIN U. Transfer learning of BCI using CUR algorithm [J]. Journal of Signal Processing Systems, 2020, 92(1): 109-121.
[45] ADAIR J, BROWNLEE A, DAOLIO F, et al. Evolving training sets for improved transfer learning in brain computer interfaces [M]//Machine learning, optimization, and big data. Cham: Springer, 2018: 186-197.
[46] DAGOIS E, KHALAF A, SEJDIC E, et al. Transfer learning for a multimodal hybrid EEG-fTCD braincomputer interface [J]. IEEE Sensors Letters, 2019, 3(1): 7100704.
[47] KHALAF A, AKCAKAYA M. A probabilistic approach for calibration time reduction in hybrid EEGfTCD brain-computer interfaces [J]. Biomedical Engineering Online, 2020, 19(1): 23.
[48] SYBELDON M, SCHMIT L, AKCAKAYA M. Transfer learning for SSVEP electroencephalography based brain–computer interfaces using Learn++.NSE and mutual information [J]. Entropy, 2017, 19(1): 41.
[49] NAKANISHI M, WANG Y T, WEI C S, et al. Facilitating calibration in high-speed BCI spellers via leveraging cross-device shared latent responses [J]. IEEE Transactions on Biomedical Engineering, 2020, 67(4): 1105-1113.
[50] BAMDADIAN A, GUAN C T, ANG K K, et al. Improving session-to-session transfer performance of motor imagery-based BCI using Adaptive Extreme Learning Machine [C]//2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Osaka: IEEE, 2013: 2188-2191.
[51] VIDAURRE C, KAWANABE M, VON B¨UNAU P, et al. Toward unsupervised adaptation of LDA for brain-computer interfaces [J]. IEEE Transactions on Biomedical Engineering, 2011, 58(3): 587-597.
[52] SP¨ULER M, ROSENSTIEL W, BOGDAN M. Adaptive SVM-based classification increases performance of a MEG-based brain-computer interface (BCI) [M]//Artificial neural networks and machine learning - ICANN 2012. Berlin, Heidelberg: Springer, 2012: 669- 676.
[53] BRUZZONE L, MARCONCINI M. Domain adaptation problems: A DASVM classification technique and a circular validation strategy [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(5): 770-787.
[54] TU W T, SUN S L. A subject transfer framework for EEG classification [J]. Neurocomputing, 2012, 82: 109-116.
[55] DALHOUMI S, DRAY G, MONTMAIN J, et al. An adaptive accuracy-weighted ensemble for intersubjects classification in brain-computer interfacing [C]//7th International IEEE EMBS Neural Engineering Conference. Montpellier: IEEE, 2015: 126-129.
[56] SALAMI A, KHODABAKHSHI M B, MORADI M H. Fuzzy transfer learning approach for analysing imagery BCI tasks [C]//2017 Artificial Intelligence and Signal Processing Conference. Shiraz: IEEE, 2017: 300-305.
[57] XU G W, SHEN X A, CHEN S R, et al. A deep transfer convolutional neural network framework for EEG signal classification [J]. IEEE Access, 2019, 7: 112767- 112776.
[58] HAJINOROOZI M, MAO Z J, LIN Y P, et al. Deep transfer learning for cross-subject and crossexperiment prediction of image rapid serial visual presentation events from EEG data [M]/ Augmented cognition neurocognition and machine learning. Cham: Springer, 2017: 45-55.
[59] ZHAO D Y, TANG F Z, SI B L, et al. Learning joint space-time-frequency features for EEG decoding on small labeled data [J]. Neural Networks, 2019, 114: 67-77.
[60] FARAHAT A, REICHERT C, SWEENEY-REED C M, et al. Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization [J]. Journal of Neural Engineering, 2019, 16(6): 066010.
[61] FAHIMI F, ZHANG Z, GOH W B, et al. Inter-subject transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI [J]. Journal of Neural Engineering, 2019, 16(2): 026007.
[62] ¨OZDENIZCI O, WANG Y, KOIKE-AKINO T, et al. Transfer learning in brain-computer interfaces with adversarial variational autoencoders [C]//2019 9th International IEEE/EMBS Conference on Neural Engineering. San Francisco, CA: IEEE, 2019: 207-210.
[63] TU W T, SUN S L. Transferable discriminative dimensionality reduction [C]//2011 IEEE 23rd International Conference on Tools with Artificial Intelligence. Boca Raton, FL: IEEE, 2011: 865-868.
[64] LEE H, CHOI S. Group nonnegative matrix factorization for EEG classification [C]//12th International Conference on Artificial Intelligence and Statistics. Clearwater Beach, FL: PMLR, 2009: 320-327.
[65] SP¨ULER M, ROSENSTIEL W, BOGDAN M. Principal component based covariate shift adaption to reduce non-stationarity in a MEG-based brain-computer interface [J]. EURASIP Journal on Advances in Signal Processing, 2012, 2012: 129.
[66] WANG P T, LU J, LU C, et al. An algorithm for movement related potentials feature extraction based on transfer learning [C]//2015 5th International Conference on Information Science and Technology. Changsha: IEEE, 2015: 309-314.
[67] WAYTOWICH N R, LAWHERN V J, BOHANNON A W, et al. Spectral transfer learning using information geometry for a user-independent brain-computer interface [J]. Frontiers in Neuroscience, 2016, 10: 430.
[68] SAMEK W, MEINECKE F C, MULLER K R. Transferring subspaces between subjects in brain: Computer interfacing [J]. IEEE Transactions on Biomedical Engineering, 2013, 60(8): 2289-2298.
[69] KANG H, NAM Y, CHOI S. Composite common spatial pattern for subject-to-subject transfer [J]. IEEE Signal Processing Letters, 2009, 16(8): 683-686.
[70] LU H P, ENG H L, GUAN C T, et al. Regularized common spatial pattern with aggregation for EEG classifi- cation in small-sample setting [J]. IEEE Transactions on Biomedical Engineering, 2010, 57(12): 2936-2946.
[71] DEVLAMINCK D, WYNS B, GROSSE-WENTRUP M, et al. Multisubject learning for common spatial patterns in motor-imagery BCI [J]. Computational Intelligence and Neuroscience, 2011, 2011: 217987.
[72] ZHU X Y, LI P Y, LI C B, et al. Separated channel convolutional neural network to realize the training free motor imagery BCI systems [J]. Biomedical Signal Processing and Control, 2019, 49: 396-403.
[73] DAI M X, ZHENG D Z, LIU S C, et al. Transfer kernel common spatial patterns for motor imagery brain-computer interface classification [J]. Computational and Mathematical Methods in Medicine, 2018, 2018: 9871603.
[74] LONG M S, WANG J M, SUN J G, et al. Domain invariant transfer kernel learning [J]. IEEE Transactions on Knowledge and Data Engineering, 2015, 27(6): 1519-1532.
[75] ZANINI P, CONGEDO M, JUTTEN C, et al. Transfer learning: A Riemannian geometry framework with applications to brain-computer interfaces [J]. IEEE Transactions on Biomedical Engineering, 2018, 65(5): 1107-1116.
[76] JU C, GAO D S, MANE R, et al. Federated transfer learning for EEG signal classification [C]//2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. Montreal, QC: IEEE, 2020: 3040-3045.
[77] RODRIGUES P L C, JUTTEN C, CONGEDO M. Riemannian Procrustes analysis: Transfer learning for brain-computer interfaces [J]. IEEE Transactions on Biomedical Engineering, 2019, 66(8): 2390-2401.
[78] GAUR P, MCCREADIE K, PACHORI R B, et al. Tangent space features-based transfer learning classification model for two-class motor imagery braincomputer interface [J]. International Journal of Neural Systems, 2019, 29(10): 1950025.
[79] ZHANG W, WU D R. Manifold embedded knowledge transfer for brain-computer interfaces [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(5): 1117-1127.
[80] MITROVIC N, ASIF M T, RASHEED U, et al. CUR decomposition for compression and compressed sensing of large-scale traffic data [C]//16th International IEEE Conference on Intelligent Transportation Systems. The Hague: IEEE, 2013: 1475-1480.
[81] WANG P T, LU J, ZHANG B, et al. A review on transfer learning for brain-computer interface classification [C]//2015 5th International Conference on Information Science and Technology. Changsha: IEEE, 2015: 315-322.
[82] LI G, LEE C H, JUNG J J, et al. Deep learning for EEG data analytics: A survey [J]. Concurrency and Computation: Practice and Experience, 2020, 32(18): e5199.
[83] YOSINSKI J, CLUNE J, BENGIO Y, et al. How transferable are features in deep neural networks? [M]//Advances in neural information processing systems 27 (NIPS 2014).Red Hook: Curran Associates, 2014: 1-9.
[84] V¨OLKER M, SCHIRRMEISTER R T, FIEDERER L D J, et al. Deep transfer learning for error decoding from non-invasive EEG [C]//2018 6th International Conference on Brain-Computer Interface. Gangwon: IEEE, 2018: 1-6.
[85] LI F, XIA Y, WANG F, et al. Transfer learning algorithm of P300-EEG signal based on XDAWN spatial filter and Riemannian geometry classifier [J]. Applied Sciences, 2020, 10(5): 1804.
[86] LEEB R, BRUNNER C, MULLER-PUTZ G R, et al. BCI Competition 2008: Graz data set B [EB/OL]. [2021-03-29]. https:// lampx.tugraz.at/～bci/ database/ 004-2014/ description.pdf.
[87] BRUNNER C, LEEB R, MULLER-PUTZ G R, et al. BCl Competition 2008: Graz data set A [EB/OL]. [2021-03-29]. https://www.bbci.de/competition/iv/desc 2a.pdf.
[88] BLANKERTZ B, DORNHEGE G, KRAULEDAT M, et al. The non-invasive Berlin Brain-Computer Interface: Fast acquisition of effective performance in untrained subjects [J]. NeuroImage, 2007, 37(2): 539-550.
[89] DORNHEGE G, BLANKERTZ B, CURIO G, et al. Boosting bit rates in noninvasive EEG single-trial classifications by feature combination and multiclass paradigms [J]. IEEE Transactions on Biomedical Engineering, 2004, 51(6): 993-1002.
[90] LEMM S, SCHAFER C, CURIO G. BCI competition 2003-data set III: Probabilistic modeling of sensorimotor/spl mu/rhythms for classification of imaginary hand movements [J]. IEEE Transactions on Biomedical Engineering, 2004, 51(6): 1077-1080.
[91] LAL T N, HINTERBERGER T, WIDMAN G, et al. Methods towards invasive human brain computer interfaces [M]//Advances in neural information processing systems 17 (NIPS 2004). Red Hook: Curran Associates, 2004: 1-8.
[92] BLANKERTZ B, CURIO G, MULLER K. R. Classifying single trial EEG: Towards brain computer interfacing [M]//Advances in neural information processing systems 14 (NIPS 2001). Red Hook: Curran Associates, 2001: 1-8.
[93] ARVANEH M, GUAN C T, ANG K K, et al. Facilitating motor imagery-based brain-computer interface for stroke patients using passive movement [J]. Neural Computing & Applications, 2017, 28(11): 3259-3272.
[94] SCHALK G, MCFARLAND D J, HINTERBERGER T, et al. BCI2000: a general-purpose brain-computer interface (BCI) system [J]. IEEE Transactions on Biomedical Engineering, 2004, 51(6): 1034-1043.