Work on instruction-tuned Large Language Models (LLMs) has used automatic
methods based on text overlap and LLM judgments as cost-effective alternatives
to human evaluation. In this paper, we study the reliability of such methods
across a broad range of tasks and in a cross-lingual setting. In contrast to
previous findings, we observe considerable variability in correlations between
automatic methods and human evaluators when scores are differentiated by task
type. Specifically, the widely-used ROUGE-L metric strongly correlates with
human judgments for short-answer English tasks but is unreliable in free-form
generation tasks and cross-lingual transfer. The effectiveness of GPT-4 as an
evaluator depends on including reference answers when prompting for
assessments, which can lead to overly strict evaluations in free-form
generation tasks. In summary, we find that, while automatic evaluation methods
can approximate human judgements under specific conditions, their reliability
is highly context-dependent. Our findings enhance the understanding of how
automatic methods should be applied and interpreted when developing and
evaluating instruction-tuned LLMs.

我们对基于文本重叠和大型语言模型判断的自动化方法在广泛任务和跨语言环境中的可靠性进行了研究，发现自动评估方法与人类评估者之间的相关性在任务类型不同的情况下存在相当大的变异性。尽管自动评估方法在特定条件下可以近似人类判断，但其可靠性高度依赖于上下文。这些发现强化了我们在开发和评估面向指导的大型语言模型时如何应用和解释自动化方法的理解。

自动评估方法在面向指导型语言模型中的可靠性研究

How Reliable Are Automatic Evaluation Methods for Instruction-Tuned  LLMs?

Transfer learning enables to re-use knowledge learned on a source task to
help learning a target task. A simple form of transfer learning is common in
current state-of-the-art computer vision models, i.e. pre-training a model for
image classification on the ILSVRC dataset, and then fine-tune on any target
task. However, previous systematic studies of transfer learning have been
limited and the circumstances in which it is expected to work are not fully
understood. In this paper we carry out an extensive experimental exploration of
transfer learning across vastly different image domains (consumer photos,
autonomous driving, aerial imagery, underwater, indoor scenes, synthetic,
close-ups) and task types (semantic segmentation, object detection, depth
estimation, keypoint detection). Importantly, these are all complex, structured
output tasks types relevant to modern computer vision applications. In total we
carry out over 2000 transfer learning experiments, including many where the
source and target come from different image domains, task types, or both. We
systematically analyze these experiments to understand the impact of image
domain, task type, and dataset size on transfer learning performance. Our study
leads to several insights and concrete recommendations: (1) for most tasks
there exists a source which significantly outperforms ILSVRC'12 pre-training;
(2) the image domain is the most important factor for achieving positive
transfer; (3) the source dataset should \emph{include} the image domain of the
target dataset to achieve best results; (4) at the same time, we observe only
small negative effects when the image domain of the source task is much broader
than that of the target; (5) transfer across task types can be beneficial, but
its success is heavily dependent on both the source and target task types.

本文通过对多种不同的图像领域和任务类型进行超过 2000 个转移学习实验，并系统地分析其对转移学习性能的影响，得出了以下几点结论：（1）对于大多数任务，存在一种源任务显著优于 ILSVRC'12 预训练；（2）图像领域是实现积极转移的最重要因素；（3）源数据集应包括目标数据集的图像领域以获得最佳结果；（4）同时，当源任务的图像领域比目标任务的图像领域范围更广时，我们观察到的负面影响很小；（5）跨任务类型的转移可能会有益，但其成功与源任务和目标任务类型密切相关。