The rapidly evolving multimodal Large Language Models (LLMs) urgently require
new benchmarks to uniformly evaluate their performance on understanding and
textually describing music. However, due to semantic gaps between Music
Information Retrieval (MIR) algorithms and human understanding, discrepancies
between professionals and the public, and low precision of annotations,
existing music description datasets cannot serve as benchmarks. To this end, we
present MuChin, the first open-source music description benchmark in Chinese
colloquial language, designed to evaluate the performance of multimodal LLMs in
understanding and describing music. We established the Caichong Music
Annotation Platform (CaiMAP) that employs an innovative multi-person,
multi-stage assurance method, and recruited both amateurs and professionals to
ensure the precision of annotations and alignment with popular semantics.
Utilizing this method, we built a dataset with multi-dimensional,
high-precision music annotations, the Caichong Music Dataset (CaiMD), and
carefully selected 1,000 high-quality entries to serve as the test set for
MuChin. Based on MuChin, we analyzed the discrepancies between professionals
and amateurs in terms of music description, and empirically demonstrated the
effectiveness of annotated data for fine-tuning LLMs. Ultimately, we employed
MuChin to evaluate existing music understanding models on their ability to
provide colloquial descriptions of music. All data related to the benchmark and
the code for scoring have been open-sourced.

MuChin 是第一个用于评估多模态大型语言模型在理解和描述音乐方面性能的中文口语音乐描述基准，利用创新的多人、多阶段保障方法和多维、高精度的音乐注释数据集 (CaiMD)，使用 MuChin 评估现有音乐理解模型对于提供口语音乐描述的能力，并开源相关数据和评分代码。