Ensuring that large language models (LLMs) reflect diverse user values and
preferences is crucial as their user bases expand globally. It is therefore
encouraging to see the growing interest in LLM personalization within the
research community. However, current works often rely on the LLM-as-a-Judge
approach for evaluation without thoroughly examining its validity. In this
paper, we investigate the reliability of LLM-as-a-Personalized-Judge, asking
LLMs to judge user preferences based on personas. Our findings suggest that
directly applying LLM-as-a-Personalized-Judge is less reliable than previously
assumed, showing low and inconsistent agreement with human ground truth. The
personas typically used are often overly simplistic, resulting in low
predictive power. To address these issues, we introduce verbal uncertainty
estimation into the LLM-as-a-Personalized-Judge pipeline, allowing the model to
express low confidence on uncertain judgments. This adjustment leads to much
higher agreement (above 80%) on high-certainty samples for binary tasks.
Through human evaluation, we find that the LLM-as-a-Personalized-Judge achieves
comparable performance to third-party humans evaluation and even surpasses
human performance on high-certainty samples. Our work indicates that
certainty-enhanced LLM-as-a-Personalized-Judge offers a promising direction for
developing more reliable and scalable methods for evaluating LLM
personalization.

基于个性化评判的高可信度大型语言模型在评估上提供了更可靠和可扩展的方法，并显示出与人类判断相当甚至超出人类在高可信度样本上的表现。