Large language models (LLMs) have revolutionized the field of natural
language processing, extending their strong capabilities into multi-modal
domains. Thus, it is vital to define proper and diversified metrics for the
evaluation of LLMs.
In this paper, we introduce matrix entropy, a novel metric rooted in
information theory and geometry principles to quantify the data compression
proficiency in LLMs. It reflects the model's ability to extract relevant
information and eliminate unnecessary elements, thereby providing insight into
the language model's intrinsic capability. Specifically, we demonstrate its
applicability in both single-modal (language) and multi-modal settings. For
language models, our findings reveal that the matrix entropy of representations
follows a scaling law type reduction when the model scales up, serving as a
complement to the traditional loss scaling law. For the multi-modal setting, we
also propose an evaluation method based on matrix entropy for assessing
alignment quality and we find that modern large multi-modal models exhibit
great alignment performance.

通过信息论和几何原理，我们引入了矩阵熵作为一种新的度量标准，以量化大型语言模型中的数据压缩能力，展示其在单模态和多模态设置中的适用性，并发现它可以作为传统损失缩放定律的补充。同时，我们基于矩阵熵提出了一种评估方法，用于评估对齐质量，并发现现代大型多模态模型表现出良好的对齐性能。