Since the breakthrough of ChatGPT, large language models (LLMs) have garnered
significant attention in the research community. With the development of LLMs,
the question of text style transfer for conversational models has emerged as a
natural extension, where chatbots may possess their own styles or even
characters. However, standard evaluation metrics have not yet been established
for this new settings. This paper aims to address this issue by proposing the
LMStyle Benchmark, a novel evaluation framework applicable to chat-style text
style transfer (C-TST), that can measure the quality of style transfer for LLMs
in an automated and scalable manner. In addition to conventional style strength
metrics, LMStyle Benchmark further considers a novel aspect of metrics called
appropriateness, a high-level metrics take account of coherence, fluency and
other implicit factors without the aid of reference samples. Our experiments
demonstrate that the new evaluation methods introduced by LMStyle Benchmark
have a higher correlation with human judgments in terms of appropriateness.
Based on LMStyle Benchmark, we present a comprehensive list of evaluation
results for popular LLMs, including LLaMA, Alpaca, and Vicuna, reflecting their
stylistic properties, such as formality and sentiment strength, along with
their appropriateness.

本研究提出了 LMStyle Benchmark，这是一个适用于聊天风格文本风格转换的评估框架，可以自动化和可扩展地评估大型语言模型的风格转换质量，并引入了适应度等新颖的度量方法，从而更好地反映人类判断。