Large language models (LLMs) that have been trained on a corpus that includes
large amount of code exhibit a remarkable ability to understand HTML code. As
web interfaces are primarily constructed using HTML, we design an in-depth
study to see how LLMs can be used to retrieve and locate important elements for
a user given query (i.e. task description) in a web interface. In contrast with
prior works, which primarily focused on autonomous web navigation, we decompose
the problem as an even atomic operation - Can LLMs identify the important
information in the web page for a user given query? This decomposition enables
us to scrutinize the current capabilities of LLMs and uncover the opportunities
and challenges they present. Our empirical experiments show that while LLMs
exhibit a reasonable level of performance in retrieving important UI elements,
there is still a substantial room for improvement. We hope our investigation
will inspire follow-up works in overcoming the current challenges in this
domain.

大型语言模型（LLM）在训练时使用了大量的代码，表现出了对 HTML 代码的卓越理解能力。因为 Web 界面主要使用 HTML 构建，所以我们进行了深入研究，探讨 LLM 如何用于在 Web 界面中检索和定位用户查询（即任务描述）的重要元素。与以往主要侧重于自主 Web 导航的研究不同，我们将问题分解为一个更为原子化的操作 - LLM 能否识别 Web 页面中对用户查询重要的信息？这种分解使我们能够审视 LLM 的当前能力并揭示其所带来的机遇和挑战。我们的实证实验证明，虽然 LLM 在检索重要的用户界面元素方面表现出了合理的性能水平，但仍然有很大的改进空间。我们希望我们的研究能够激发在这一领域克服当前挑战的后续工作。

使用 LLMs 从 Web 界面检索信息的机会与挑战

"What's important here?": Opportunities and Challenges of Using LLMs in  Retrieving Information from Web Interfaces

Translating natural language utterances to executable queries is a helpful
technique in making the vast amount of data stored in relational databases
accessible to a wider range of non-tech-savvy end users. Prior work in this
area has largely focused on textual input that is linguistically correct and
semantically unambiguous. However, real-world user queries are often succinct,
colloquial, and noisy, resembling the input of a search engine. In this work,
we introduce data augmentation techniques and a sampling-based content-aware
BERT model (ColloQL) to achieve robust text-to-SQL modeling over natural
language search (NLS) questions. Due to the lack of evaluation data, we curate
a new dataset of NLS questions and demonstrate the efficacy of our approach.
ColloQL's superior performance extends to well-formed text, achieving 84.9%
(logical) and 90.7% (execution) accuracy on the WikiSQL dataset, making it, to
the best of our knowledge, the highest performing model that does not use
execution guided decoding.

使用数据增强技术和基于采样的内容感知 BERT 模型 (ColloQL) 实现了鲁棒的自然语言搜索 (NLS) 查询的文本到 SQL 建模，评估表明该方法优于现有技术。

ColloQL：搜索查询的跨领域文本转 SQL 的鲁棒性解决方案

ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries

Video summarization plays an important role in video understanding by
selecting key frames/shots. Traditionally, it aims to find the most
representative and diverse contents in a video as short summaries. Recently, a
more generalized task, query-conditioned video summarization, has been
introduced, which takes user queries into consideration to learn more
user-oriented summaries. In this paper, we propose a query-conditioned
three-player generative adversarial network to tackle this challenge. The
generator learns the joint representation of the user query and the video
content, and the discriminator takes three pairs of query-conditioned summaries
as the input to discriminate the real summary from a generated and a random
one. A three-player loss is introduced for joint training of the generator and
the discriminator, which forces the generator to learn better summary results,
and avoids the generation of random trivial summaries. Experiments on a
recently proposed query-conditioned video summarization benchmark dataset show
the efficiency and efficacy of our proposed method.

本文提出了一种查询驱动的三方生成对抗网络，通过学习用户查询和视频内容的联合表示，并引入三方损失函数，即鼓励生成器学习更好的摘要结果，避免生成随机无意义摘要。实验结果表明，该方法在一些查询驱动的视频摘要基准数据集上表现出了高效性和可行性。