Speech decoding from EEG signals is a challenging task, where brain activity
is modeled to estimate salient characteristics of acoustic stimuli. We propose
FESDE, a novel framework for Fully-End-to-end Speech Decoding from EEG signals.
Our approach aims to directly reconstruct listened speech waveforms given EEG
signals, where no intermediate acoustic feature processing step is required.
The proposed method consists of an EEG module and a speech module along with a
connector. The EEG module learns to better represent EEG signals, while the
speech module generates speech waveforms from model representations. The
connector learns to bridge the distributions of the latent spaces of EEG and
speech. The proposed framework is both simple and efficient, by allowing
single-step inference, and outperforms prior works on objective metrics. A
fine-grained phoneme analysis is conducted to unveil model characteristics of
speech decoding. The source code is available here: github.com/lee-jhwn/fesde.

EEG 信号的完全端到端语音解码新框架 FESDE，通过学习和连接 EEG 信号的潜在空间和语音波形，直接重构听到的语音波形，无需中间的声学特征处理步骤，并在客观度量指标上超过了以前的方法。