The process of reconstructing experiences from human brain activity offers a
unique lens into how the brain interprets and represents the world. In this
paper, we introduce a method for reconstructing music from brain activity,
captured using functional magnetic resonance imaging (fMRI). Our approach uses
either music retrieval or the MusicLM music generation model conditioned on
embeddings derived from fMRI data. The generated music resembles the musical
stimuli that human subjects experienced, with respect to semantic properties
like genre, instrumentation, and mood. We investigate the relationship between
different components of MusicLM and brain activity through a voxel-wise
encoding modeling analysis. Furthermore, we discuss which brain regions
represent information derived from purely textual descriptions of music
stimuli. We provide supplementary material including examples of the
reconstructed music at this https URL

通过功能性磁共振成像（fMRI）捕获的脑活动，我们介绍了一种从脑活动重构音乐的方法，该方法使用音乐检索或 MusicLM 音乐生成模型，根据 fMRI 数据的嵌入来生成类似于人类主体经历的音乐，包括语义特征如流派、乐器和情绪。我们通过体素编码模型分析，探讨了 MusicLM 的不同组件与脑活动之间的关系，并讨论了哪些脑区表示来自纯文本音乐描述的信息。我们提供了补充材料，包括重构音乐的示例。

Brain2Music: 从人类脑电活动重构音乐

Brain2Music: Reconstructing Music from Human Brain Activity

Music similarity is an essential aspect of music retrieval, recommendation
systems, and music analysis. Moreover, similarity is of vital interest for
music experts, as it allows studying analogies and influences among composers
and historical periods. Current approaches to musical similarity rely mainly on
symbolic content, which can be expensive to produce and is not always readily
available. Conversely, approaches using audio signals typically fail to provide
any insight about the reasons behind the observed similarity. This research
addresses the limitations of current approaches by focusing on the study of
musical similarity using both symbolic and audio content. The aim of this
research is to develop a fully explainable and interpretable system that can
provide end-users with more control and understanding of music similarity and
classification systems.

本研究旨在开发一种可解释的音乐相似性分类系统，通过结合符号和音频内容，提高音乐检索、推荐和分析的效率和精度。

基于知识的多模式音乐相似度

Knowledge-based Multimodal Music Similarity

Lyric interpretations can help people understand songs and their lyrics
quickly, and can also make it easier to manage, retrieve and discover songs
efficiently from the growing mass of music archives. In this paper we propose
BART-fusion, a novel model for generating lyric interpretations from lyrics and
music audio that combines a large-scale pre-trained language model with an
audio encoder. We employ a cross-modal attention module to incorporate the
audio representation into the lyrics representation to help the pre-trained
language model understand the song from an audio perspective, while preserving
the language model's original generative performance. We also release the Song
Interpretation Dataset, a new large-scale dataset for training and evaluating
our model. Experimental results show that the additional audio information
helps our model to understand words and music better, and to generate precise
and fluent interpretations. An additional experiment on cross-modal music
retrieval shows that interpretations generated by BART-fusion can also help
people retrieve music more accurately than with the original BART.

本文提出了 BART-fusion 模型，用于从音乐音频和歌词中生成歌词解释，该模型结合了一个大规模预训练语言模型和一个音频编码器。实验结果显示，该模型能够更好地理解歌词和音乐，生成精确流畅的解释，并且可以帮助人们更准确地检索音乐。

使用音频指导的预训练语言模型解释歌曲歌词

Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model

This paper demonstrates the feasibility of learning to retrieve short
snippets of sheet music (images) when given a short query excerpt of music
(audio) -- and vice versa --, without any symbolic representation of music or
scores. This would be highly useful in many content-based musical retrieval
scenarios. Our approach is based on Deep Canonical Correlation Analysis (DCCA)
and learns correlated latent spaces allowing for cross-modality retrieval in
both directions. Initial experiments with relatively simple monophonic music
show promising results.

本文旨在研究跨媒体检索中的乐曲检索，基于 Deep Canonical Correlation Analysis 学习相关空间，能够检索图片和音频之间的乐曲片段，初步实验结果有所进展。