This research investigates prompt designs of evaluating generated texts using
large language models (LLMs). While LLMs are increasingly used for scoring
various inputs, creating effective prompts for open-ended text evaluation
remains challenging due to model sensitivity and subjectivity in evaluation of
text generation. Our study experimented with different prompt structures,
altering the sequence of output instructions and including explanatory reasons.
We found that the order of presenting reasons and scores significantly
influences LLMs' scoring, with a different level of rule understanding in the
prompt. An additional optimization may enhance scoring alignment if sufficient
data is available. This insight is crucial for improving the accuracy and
consistency of LLM-based evaluations.

通过研究大型语言模型的评估生成文本的提示设计，本研究发现不同提示结构和包含解释性原因的顺序对语言模型评分有重要影响，进而提出了优化评分一致性的方法。

文本生成的更好 LLM 评估器：提示输出排序和优化的影响

A Better LLM Evaluator for Text Generation: The Impact of Prompt Output  Sequencing and Optimization

Recursion is one of the hallmarks of human language. While many design
features of language have been shown to exist in animal communication systems,
recursion has not. Previous research shows that GPT-4 is the first large
language model (LLM) to exhibit metalinguistic abilities (Begu\v{s},
D\k{a}bkowski, and Rhodes 2023). Here, we propose several prompt designs aimed
at eliciting and analyzing recursive behavior in LLMs, both linguistic and
non-linguistic. We demonstrate that when explicitly prompted, GPT-4 can both
produce and analyze recursive structures. Thus, we present one of the first
studies investigating whether meta-linguistic awareness of recursion -- a
uniquely human cognitive property -- can emerge in transformers with a high
number of parameters such as GPT-4.

本研究旨在利用多种提示设计来唤起和分析 LLMs 中的递归行为，同时论证了 GPT-4 可以在明确提示的情况下产生和分析递归结构，也是第一次研究语言模型中类元语言意识是否会出现的研究之一。