In this work, we introduce a novel approach that equips LLM agents with
introspection, enhancing consistency and adaptability in solving complex tasks.
Our approach prompts LLM agents to decompose a given task into manageable
subtasks (i.e., to make a plan), and to continuously introspect upon the
suitability and results of their actions. We implement a three-fold
introspective intervention: 1) anticipatory reflection on potential failures
and alternative remedy before action execution, 2) post-action alignment with
subtask objectives and backtracking with remedy to ensure utmost effort in plan
execution, and 3) comprehensive review upon plan completion for future strategy
refinement. By deploying and experimenting with this methodology - a zero-shot
approach - within WebArena for practical tasks in web environments, our agent
demonstrates superior performance over existing zero-shot methods. The
experimental results suggest that our introspection-driven approach not only
enhances the agent's ability to navigate unanticipated challenges through a
robust mechanism of plan execution, but also improves efficiency by reducing
the number of trials and plan revisions needed to achieve a task.

我们介绍了一种新颖的方法，为 LLM 智能体提供自省能力，以增强其在解决复杂任务中的一致性和适应性。我们的方法促使 LLM 智能体将给定的任务分解为可管理的子任务（即制定计划），并持续对其行动的适用性和结果进行自省。通过在 Web 环境中使用这种零样本方法进行实际任务的部署和实验，我们的智能体展示了优于现有零样本方法的性能。实验结果表明，我们的自省驱动方法不仅通过稳定的计划执行机制增强了智能体应对意外挑战的能力，而且通过减少试验和计划修订所需的次数，提高了任务的效率。

魔鬼的辩护：面向 LLM 代理的预期反思

Devil's Advocate: Anticipatory Reflection for LLM Agents

The advent of immersive Virtual Reality applications has transformed various
domains, yet their integration with advanced artificial intelligence
technologies like Visual Language Models remains underexplored. This study
introduces a pioneering approach utilizing VLMs within VR environments to
enhance user interaction and task efficiency. Leveraging the Unity engine and a
custom-developed VLM, our system facilitates real-time, intuitive user
interactions through natural language processing, without relying on visual
text instructions. The incorporation of speech-to-text and text-to-speech
technologies allows for seamless communication between the user and the VLM,
enabling the system to guide users through complex tasks effectively.
Preliminary experimental results indicate that utilizing VLMs not only reduces
task completion times but also improves user comfort and task engagement
compared to traditional VR interaction methods.

本研究介绍了一种创新方法，利用 VR 环境中的 VLMs（Visual Language Models）来增强用户交互和任务效率，通过自然语言处理实现实时、直观的用户交互，不依赖于视觉文本指令。使用语音识别和文本转语音技术，使用户与 VLM 之间实现无缝通信，有效地引导用户完成复杂任务。初步实验结果显示，使用 VLMs 不仅能够缩短任务完成时间，还能提高用户舒适度和任务参与度，相较于传统的 VR 交互方法。

VR-GPT：智能虚拟现实应用的视觉语言模型

VR-GPT: Visual Language Model for Intelligent Virtual Reality  Applications

Human-robot collaboration (HRC) is one key component to achieving flexible
manufacturing to meet the different needs of customers. However, it is
difficult to build intelligent robots that can proactively assist humans in a
safe and efficient way due to several challenges.First, it is challenging to
achieve efficient collaboration due to diverse human behaviors and data
scarcity. Second, it is difficult to ensure interactive safety due to
uncertainty in human behaviors. This paper presents an integrated framework for
proactive HRC. A robust intention prediction module, which leverages prior task
information and human-in-the-loop training, is learned to guide the robot for
efficient collaboration. The proposed framework also uses robust safe control
to ensure interactive safety under uncertainty. The developed framework is
applied to a co-assembly task using a Kinova Gen3 robot. The experiment
demonstrates that our solution is robust to environmental changes as well as
different human preferences and behaviors. In addition, it improves task
efficiency by approximately 15-20%. Moreover, the experiment demonstrates that
our solution can guarantee interactive safety during proactive collaboration.

本文介绍了一种集成框架，用于支持主动的人机协作，并解决了数据稀缺、人员行为多样、交互安全不确定等问题。实验表明，该框架在 Kinova Gen3 机器人的组装任务中，能有效地提高任务效率，以及确保主动协作的安全性。