We introduce MuseChat, an innovative dialog-based music recommendation system. This unique platform not only offers interactive user engagement but also suggests music tailored for input videos, so that users can refine and personalize their music selections. In contrast, previous systems predominantly emphasized content compatibility, often overlooking the nuances of users' individual preferences. For example, all the datasets only provide basic music-video pairings or such pairings with textual music descriptions. To address this gap, our research offers three contributions. First, we devise a conversation-synthesis method that simulates a two-turn interaction between a user and a recommendation system, which leverages pre-trained music tags and artist information. In this interaction, users submit a video to the system, which then suggests a suitable music piece with a rationale. Afterwards, users communicate their musical preferences, and the system presents a refined music recommendation with reasoning. Second, we introduce a multi-modal recommendation engine that matches music either by aligning it with visual cues from the video or by harmonizing visual information, feedback from previously recommended music, and the user's textual input. Third, we bridge music representations and textual data with a Large Language Model(Vicuna-7B). This alignment equips MuseChat to deliver music recommendations and their underlying reasoning in a manner resembling human communication. Our evaluations show that MuseChat surpasses existing state-of-the-art models in music retrieval tasks and pioneers the integration of the recommendation process within a natural language framework.

MuseChat是一个创新的对话式音乐推荐系统，通过模拟用户与推荐系统之间的互动，利用预训练的音乐标签和艺术家信息，将音乐推荐过程融合到自然语言框架中，以人类交流方式提供音乐推荐和推荐原因，以图像和文本搭配的多模态推荐引擎，使音乐匹配音视频中的视觉线索或协调视觉信息、之前推荐音乐的反馈和用户的文本输入，MuseChat在音乐检索任务中超过现有的最先进模型，是推荐过程与自然语言框架相结合的先驱。

MuseChat: 视频的对话式音乐推荐系统