Decoding inner speech from the brain signal via hybridisation of fMRI and EEG
data is explored to investigate the performance benefits over unimodal models.
Two different bimodal fusion approaches are examined: concatenation of
probability vectors output from unimodal fMRI and EEG machine learning models,
and data fusion with feature engineering. Same task inner speech data are
recorded from four participants, and different processing strategies are
compared and contrasted to previously-employed hybridisation methods. Data
across participants are discovered to encode different underlying structures,
which results in varying decoding performances between subject-dependent fusion
models. Decoding performance is demonstrated as improved when pursuing bimodal
fMRI-EEG fusion strategies, if the data show underlying structure.

本文利用融合功能磁共振成像 (fMRI) 和脑电图 (EEG) 的方法对内部言语进行了解码并探究了其性能优势。作者对两种不同的双模融合方法进行了研究，包括从单模 fMRI 和 EEG 机器学习模型的概率向量输出和特征工程的数据融合。作者通过四位参与者记录的同一任务的内部言语数据，比较和对比了不同的处理策略并探索了不同的融合模型对理解内部言语的表现差异。研究发现不同参与者的数据编码了不同的基础结构，因此，基于受试者的融合模型的解码表现存在差异。如果数据表现出基础结构，则深入探究双模 fMRI-EEG 融合策略的解码性能有所提高。