Recent popularity of Large Language Models (LLMs) has opened countless
possibilities in automating numerous AI tasks by connecting LLMs to various
domain-specific models or APIs, where LLMs serve as dispatchers while
domain-specific models or APIs are action executors. Despite the vast numbers
of domain-specific models/APIs, they still struggle to comprehensively cover
super diverse automation demands in the interaction between human and User
Interfaces (UIs). In this work, we build a multimodal model to ground natural
language instructions in given UI screenshots as a generic UI task automation
executor. This metadata-free grounding model, consisting of a visual encoder
and a language decoder, is first pretrained on well studied document
understanding tasks and then learns to decode spatial information from UI
screenshots in a promptable way. To facilitate the exploitation of
image-to-text pretrained knowledge, we follow the pixel-to-sequence paradigm to
predict geometric coordinates in a sequence of tokens using a language decoder.
We further propose an innovative Reinforcement Learning (RL) based algorithm to
supervise the tokens in such sequence jointly with visually semantic metrics,
which effectively strengthens the spatial decoding capability of the
pixel-to-sequence paradigm. Extensive experiments demonstrate our proposed
reinforced UI instruction grounding model outperforms the state-of-the-art
methods by a clear margin and shows the potential as a generic UI task
automation API.

建立了一个多模态模型用于将自然语言指令与给定的 UI 屏幕截图联系起来，作为通用的 UI 任务自动执行器，并通过强化学习算法对其进行加强，实验结果表明，该模型在 UI 任务自动化方面表现优异，显示出作为通用 UI 任务自动化 API 的潜力。

增强型用户界面指令基础：走向通用的用户界面任务自动化 API

Reinforced UI Instruction Grounding: Towards a Generic UI Task  Automation API

In this report, we present our champion solution for Ego4D Natural Language
Queries (NLQ) Challenge in CVPR 2023. Essentially, to accurately ground in a
video, an effective egocentric feature extractor and a powerful grounding model
are required. Motivated by this, we leverage a two-stage pre-training strategy
to train egocentric feature extractors and the grounding model on video
narrations, and further fine-tune the model on annotated data. In addition, we
introduce a novel grounding model GroundNLQ, which employs a multi-modal
multi-scale grounding module for effective video and text fusion and various
temporal intervals, especially for long videos. On the blind test set,
GroundNLQ achieves 25.67 and 18.18 for R1@IoU=0.3 and R1@IoU=0.5, respectively,
and surpasses all other teams by a noticeable margin. Our code will be released
athttps://github.com/houzhijian/GroundNLQ.

本文介绍了一种基于两阶段预训练策略及新颖的 GroundNLQ 模型的视频摘要方法。在 Ego4D 自然语言查询挑战方面获得优异的表现，超过其他团队。

2023 年 GroundNLQ@Ego4D 自然语言查询挑战赛

GroundNLQ @ Ego4D Natural Language Queries Challenge 2023

The limited scale of annotated data constraints existing context-dependent
text-to-SQL models because of the complexity of labeling. The data augmentation
method is a commonly used method to solve this problem. However, the data
generated by current augmentation methods often lack diversity. In this paper,
we introduce ConDA, which generates interactive questions and corresponding SQL
results. We designed the SQL dialogue state to enhance the data diversity
through the state transition. Meanwhile, we also present a filter method to
ensure the data quality by a grounding model. Additionally, we utilize a
grounding model to identify and filter low-quality questions that mismatch the
state information. Experimental results on the SParC and CoSQL datasets show
that ConDA boosts the baseline model to achieve an average improvement of
$3.3\%$ on complex questions. Moreover, we analyze the augmented data, which
reveals that the data generated by ConDA are of high quality in both SQL
template hardness and types, turns, and question consistency.

本文介绍了一种名为 ConDA 的数据扩充方法，通过交互式问题和相应的 SQL 结果生成数据，利用 SQL 对话状态增强数据多样性，同时利用基础模型确保数据质量。实验结果表明，ConDA 使基础模型的性能平均提升了 3.3％，且生成的数据质量较高。