We show that domain-general automatic evaluators can significantly improve
the performance of agents for web navigation and device control. We experiment
with multiple evaluation models that trade off between inference cost,
modularity of design, and accuracy. We validate the performance of these models
in several popular benchmarks for digital agents, finding between 74.4 and
92.9% agreement with oracle evaluation metrics. Finally, we use these
evaluators to improve the performance of existing agents via fine-tuning and
inference-time guidance. Without any additional supervision, we improve
state-of-the-art performance by 29% on the popular benchmark WebArena, and
achieve a 75% relative improvement in a challenging domain transfer scenario.

使用通用自动评估器能够显著提高网络导航和设备控制代理的性能，并通过精确的推理成本、设计模块性以及准确性之间的权衡实验了多个评估模型；通过微调和推理时指导使用这些评估器来提高现有代理的性能，在 WebArena 受欢迎的基准测试中，不需要任何额外的监督，使性能达到了 29% 的提升，并在具有挑战性的领域转移场景中实现了 75% 的相对改进。

数字代理的自主评估和优化

Autonomous Evaluation and Refinement of Digital Agents

Much of the previous work towards digital agents for graphical user
interfaces (GUIs) has relied on text-based representations (derived from HTML
or other structured data sources), which are not always readily available.
These input representations have been often coupled with custom, task-specific
action spaces. This paper focuses on creating agents that interact with the
digital world using the same conceptual interface that humans commonly use --
via pixel-based screenshots and a generic action space corresponding to
keyboard and mouse actions. Building upon recent progress in pixel-based
pretraining, we show, for the first time, that it is possible for such agents
to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based
instruction following tasks.

本研究提出一种使用基于像素的屏幕截图和一般的鼠标和键盘动作空间来模拟人与数字世界互动的智能代理。通过依赖于最近的像素预训练技术，该代理在 GUI 任务的 MinWob ++ 基准测试中表现出比人类众包工人更好的性能。

从像素到 UI 操作：通过图形用户界面学习跟随指令

From Pixels to UI Actions: Learning to Follow Instructions via Graphical  User Interfaces

While utilization of digital agents to support crucial decision making is
increasing, trust in suggestions made by these agents is hard to achieve.
However, it is essential to profit from their application, resulting in a need
for explanations for both the decision making process and the model. For many
systems, such as common black-box models, achieving at least some
explainability requires complex post-processing, while other systems profit
from being, to a reasonable extent, inherently interpretable. We propose a
rule-based learning system specifically conceptualised and, thus, especially
suited for these scenarios. Its models are inherently transparent and easily
interpretable by design. One key innovation of our system is that the rules'
conditions and which rules compose a problem's solution are evolved separately.
We utilise independent rule fitnesses which allows users to specifically tailor
their model structure to fit the given requirements for explainability.

本文针对数字代理支持关键决策的应用难以获得信任这一难题，提出了基于规则学习的系统并设计透明易懂的模型，其中规则条件和问题解决方案的组成是分开演化的，允许用户特别定制模型以适应可解释性的要求。