Evaluating grounded neural language model performance with respect to
pragmatic qualities like the trade off between truthfulness, contrastivity and
overinformativity of generated utterances remains a challenge in absence of
data collected from humans. To enable such evaluation, we present a novel open
source image-text dataset "Annotated 3D Shapes" (A3DS) comprising over nine
million exhaustive natural language annotations and over 12 million
variable-granularity captions for the 480,000 images provided by Burges & Kim
(2018). We showcase the evaluation of pragmatic abilities developed by a
task-neutral image captioner fine-tuned in a multi-agent communication setting
to produce contrastive captions. The evaluation is enabled by the dataset
because the exhaustive annotations allow to quantify the presence of
contrastive features in the model's generations. We show that the model
develops human-like patterns (informativity, brevity, over-informativity for
specific features (e.g., shape, color biases)).

通过使用我们新开发的开源图像 - 文本数据集 “注释的 3D 形状” 来评估一个任务中性的图像描述模型的语用能力，我们发现该模型在开发对比性标题方面具有类似于人类的模式（信息量，简洁性和过度信息性）。

评估 A3DS 图像描述程序的语用能力

Evaluating Pragmatic Abilities of Image Captioners on A3DS

The extraordinary performance of large language models (LLMs) heightens the
importance of detecting whether the context is generated by an AI system. More
importantly, while more and more companies and institutions release their LLMs,
the origin can be hard to trace. Since LLMs are heading towards the time of
AGI, similar to the origin tracing in anthropology, it is of great importance
to trace the origin of LLMs. In this paper, we first raise the concern of the
origin tracing of LLMs and propose an effective method to trace and detect
AI-generated contexts. We introduce a novel algorithm that leverages the
contrastive features between LLMs and extracts model-wise features to trace the
text origins. Our proposed method works under both white-box and black-box
settings therefore can be widely generalized to detect various LLMs.(e.g. can
be generalized to detect GPT-3 models without the GPT-3 models). Also, our
proposed method requires only limited data compared with the supervised
learning methods and can be extended to trace new-coming model origins. We
construct extensive experiments to examine whether we can trace the origins of
given texts. We provide valuable observations based on the experimental
results, such as the difficulty level of AI origin tracing, and the AI origin
similarities, and call for ethical concerns of LLM providers. We are releasing
all codes and data as a toolkit and benchmark for future AI origin tracing and
detecting studies. \footnote{We are releasing all available resource at
https://github.com/OpenLMLab/.}

本文提出了一种有效追踪和检测人工智能生成上下文的方法，并介绍了一种利用对比特征提取模型特征来追踪文本来源的新算法，该算法可以适用于各种大型语言模型的检测和跟踪，并提供了关于实验结果的有价值观察，如人工智能起源的难度和人工智能起源的相似性，并呼吁对所有大型语言模型提供者的道德问题进行关注。