Current captioning datasets, focus on object-centric captions, describing the
visible objects in the image, often ending up stating the obvious (for humans),
e.g. "people eating food in a park". Although these datasets are useful to
evaluate the ability of Vision & Language models to recognize the visual
content, they lack in expressing trivial abstract concepts, e.g. "people having
a picnic". Such concepts are licensed by human's personal experience and
contribute to forming common sense assumptions. We present the High-Level
Dataset; a dataset extending 14997 images of the COCO dataset with 134973
human-annotated (high-level) abstract captions collected along three axes:
scenes, actions and rationales. We describe and release such dataset and we
show how it can be used to assess models' multimodal grounding of abstract
concepts and enrich models' visio-lingusitic representations. Moreover, we
describe potential tasks enabled by this dataset involving high- and low-level
concepts interactions.

本文介绍一个新的高级数据集（High-Level Dataset），可以拓展经典 COOC 数据集，使得机器学习模型更好地理解抽象概念，并进一步提升模型的多模态融合能力。

HL 数据集：将高层语言概念与视觉相结合

HL Dataset: Grounding High-Level Linguistic Concepts in Vision

Humans can learn to operate the user interface (UI) of an application by
reading an instruction manual or how-to guide. Along with text, these resources
include visual content such as UI screenshots and images of application icons
referenced in the text. We explore how to leverage this data to learn generic
visio-linguistic representations of UI screens and their components. These
representations are useful in many real applications, such as accessibility,
voice navigation, and task automation. Prior UI representation models rely on
UI metadata (UI trees and accessibility labels), which is often missing,
incompletely defined, or not accessible. We avoid such a dependency, and
propose Lexi, a pre-trained vision and language model designed to handle the
unique features of UI screens, including their text richness and context
sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k
UI images paired with descriptions of their functionality. We evaluate Lexi on
four tasks: UI action entailment, instruction-based UI image retrieval,
grounding referring expressions, and UI entity recognition.

本研究探索如何利用 UI 说明书中的图像来学习 UI 屏幕及其组件的通用视觉 - 语言表示，并提出了一种名为 Lexi 的预训练模型，该模型可处理 UI 屏幕的独特特点，包括其丰富的文本和上下文敏感性。通过创建包含 114k UI 图像及其功能描述的 UICaption 数据集，我们证明了该模型可用于 UI 行动蕴含和 UI 实体识别等任务。

Lexi: UI 语言的自监督学习

Lexi: Self-Supervised Learning of the UI Language

Numerous recent works have proposed pretraining generic visio-linguistic
representations and then finetuning them for downstream vision and language
tasks. While architecture and objective function design choices have received
attention, the choice of pretraining datasets has received little attention. In
this work, we question some of the default choices made in literature. For
instance, we systematically study how varying similarity between the
pretraining dataset domain (textual and visual) and the downstream domain
affects performance. Surprisingly, we show that automatically generated data in
a domain closer to the downstream task (e.g., VQA v2) is a better choice for
pretraining than "natural" data but of a slightly different domain (e.g.,
Conceptual Captions). On the other hand, some seemingly reasonable choices of
pretraining datasets were found to be entirely ineffective for some downstream
tasks. This suggests that despite the numerous recent efforts, vision &
language pretraining does not quite work "out of the box" yet. Overall, as a
by-product of our study, we find that simple design choices in pretraining can
help us achieve close to state-of-art results on downstream tasks without any
architectural changes.

该研究在研究预训练数据集的选择时发现，与下游任务相似的自动生成数据集是预训练的更好选择，而某些合理的预训练数据集对某些下游任务完全无效，这表明视听语言预训练仍需要简单的设计选择来实现接近最新成果的结果。