Drawing inspiration from the hierarchical processing of the human auditory
system, which transforms sound from low-level acoustic features to high-level
semantic understanding, we introduce a novel coarse-to-fine audio
reconstruction method. Leveraging non-invasive functional Magnetic Resonance
Imaging (fMRI) data, our approach mimics the inverse pathway of auditory
processing. Initially, we utilize CLAP to decode fMRI data coarsely into a
low-dimensional semantic space, followed by a fine-grained decoding into the
high-dimensional AudioMAE latent space guided by semantic features. These
fine-grained neural features serve as conditions for audio reconstruction
through a Latent Diffusion Model (LDM). Validation on three public fMRI
datasets-Brain2Sound, Brain2Music, and Brain2Speech-underscores the superiority
of our coarse-to-fine decoding method over stand-alone fine-grained approaches,
showcasing state-of-the-art performance in metrics like FD, FAD, and KL.
Moreover, by employing semantic prompts during decoding, we enhance the quality
of reconstructed audio when semantic features are suboptimal. The demonstrated
versatility of our model across diverse stimuli highlights its potential as a
universal brain-to-audio framework. This research contributes to the
comprehension of the human auditory system, pushing boundaries in neural
decoding and audio reconstruction methodologies.

基于人类听觉系统的层次化处理，从低级声学特征到高级语义理解的转变，我们引入了一种新的粗到细的音频重建方法。利用非侵入式功能性磁共振成像（fMRI）数据，我们的方法模拟了听觉处理的逆向路径。通过初始使用 CLAP 将 fMRI 数据粗略解码到低维语义空间，然后通过语义特征引导进行精细解码到高维 AudioMAE 潜空间。这些精细的神经特征用作通过潜在扩散模型（LDM）进行音频重建的条件。验证在三个公共 fMRI 数据集（Brain2Sound、Brain2Music 和 Brain2Speech）上，我们的粗到细解码方法相对于独立的精细解码方法具有明显优势，并展示了 FD、FAD 和 KL 等度量指标上的最先进性能。此外，在解码过程中使用语义提示时，当语义特征不佳时，我们提高了重建音频的质量。我们模型在各种刺激下的展示性能突出了作为通用脑 - 音频框架的潜力。这项研究对于理解人类听觉系统，在神经解码和音频重建方法方面推动了界限。