A common problem for agents operating in real-world environments is that the
response of an environment to their actions may be non-deterministic and
observed through noise. This renders environmental state and progress towards
completing a task latent. Despite recent impressive demonstrations of LLM's
reasoning abilities on various benchmarks, whether LLMs can build estimates of
latent state and leverage them for reasoning has not been explicitly studied.
We investigate this problem in the real-world domain of autonomous UI agents.
We establish that appropriately prompting LLMs in a zero-shot manner can be
formally understood as forming point estimates of latent state in a textual
space. In the context of autonomous UI agents we then show that LLMs used in
this manner are more than $76\%$ accurate at inferring various aspects of
latent state, such as performed (vs. commanded) actions and task progression.
Using both public and internal benchmarks and three reasoning methods
(zero-shot, CoT-SC & ReAct), we show that LLM-powered agents that explicitly
estimate and reason about latent state are able to successfully complete up to
1.6x more tasks than those that do not.

研究了代理在现实环境中的一个常见问题，即环境对其行为的响应可能是不确定的，通过噪声观察到。通过适当的提示 LLMs 以零 - shot 方式可以被形式化地理解为在文本空间中形成对潜在状态的点估计。在自主 UI 代理的背景下，我们展示了以这种方式使用 LLMs 在推断潜在状态各方面（如已执行（与已命令）的操作和任务进展）上的准确性超过 76％。使用公共和内部基准测试以及三种推理方法（零 - shot，CoT-SC 和 ReAct），我们表明，明确估计和推理潜在状态的 LLM 驱动代理比那些不这样做的代理能够成功完成多达 1.6 倍的任务。

潜在状态估计有助于用户界面代理进行推理

Latent State Estimation Helps UI Agents to Reason

Multimodal pretraining has emerged as an effective strategy for the trinity
of goals of representation learning in autonomous robots: 1) extracting both
local and global task progression information; 2) enforcing temporal
consistency of visual representation; 3) capturing trajectory-level language
grounding. Most existing methods approach these via separate objectives, which
often reach sub-optimal solutions. In this paper, we propose a universal
unified objective that can simultaneously extract meaningful task progression
information from image sequences and seamlessly align them with language
instructions. We discover that via implicit preferences, where a visual
trajectory inherently aligns better with its corresponding language instruction
than mismatched pairs, the popular Bradley-Terry model can transform into
representation learning through proper reward reparameterizations. The resulted
framework, DecisionNCE, mirrors an InfoNCE-style objective but is distinctively
tailored for decision-making tasks, providing an embodied representation
learning framework that elegantly extracts both local and global task
progression features, with temporal consistency enforced through implicit time
contrastive learning, while ensuring trajectory-level instruction grounding via
multimodal joint encoding. Evaluation on both simulated and real robots
demonstrates that DecisionNCE effectively facilitates diverse downstream policy
learning tasks, offering a versatile solution for unified representation and
reward learning. Project Page: this https URL

通过多模态预训练方法，本论文提出了一种通用的统一目标，可以同时从图像序列中提取有意义的任务进展信息，并将其与语言指令无缝对齐，以提供一种优雅地提取本地和全局任务进展特征的实体化表示学习框架，通过隐式时间对比学习强制执行时间一致性，并通过多模态联合编码确保轨迹级指令定位，从而为统一的表示和奖励学习提供了多样化的解决方案。