Evaluating grounded neural language model performance with respect to
pragmatic qualities like the trade off between truthfulness, contrastivity and
overinformativity of generated utterances remains a challenge in absence of
data collected from humans. To enable such evaluation, we present a novel open
source image-text dataset "Annotated 3D Shapes" (A3DS) comprising over nine
million exhaustive natural language annotations and over 12 million
variable-granularity captions for the 480,000 images provided by Burges & Kim
(2018). We showcase the evaluation of pragmatic abilities developed by a
task-neutral image captioner fine-tuned in a multi-agent communication setting
to produce contrastive captions. The evaluation is enabled by the dataset
because the exhaustive annotations allow to quantify the presence of
contrastive features in the model's generations. We show that the model
develops human-like patterns (informativity, brevity, over-informativity for
specific features (e.g., shape, color biases)).

通过使用我们新开发的开源图像 - 文本数据集 “注释的 3D 形状” 来评估一个任务中性的图像描述模型的语用能力，我们发现该模型在开发对比性标题方面具有类似于人类的模式（信息量，简洁性和过度信息性）。