Programming robot behaviour in a complex world faces challenges on multiple
levels, from dextrous low-level skills to high-level planning and reasoning.
Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning
ability in zero-shot robotic planning. However, it remains challenging to
ground LLMs in multimodal sensory input and continuous action output, while
enabling a robot to interact with its environment and acquire novel information
as its policies unfold. We develop a robot interaction scenario with a
partially observable state, which necessitates a robot to decide on a range of
epistemic actions in order to sample sensory information among multiple
modalities, before being able to execute the task correctly. An interactive
perception framework is therefore proposed with an LLM as its backbone, whose
ability is exploited to instruct epistemic actions and to reason over the
resulting multimodal sensations (vision, sound, haptics, proprioception), as
well as to plan an entire task execution based on the interactively acquired
information. Our study demonstrates that LLMs can provide high-level planning
and reasoning skills and control interactive robot behaviour in a multimodal
environment, while multimodal modules with the context of the environmental
state help ground the LLMs and extend their processing ability.

在机器人的交互感知中，使用预先训练的大型语言模型（LLMs）作为交互感知框架，并将其应用于决策问题以及规划多模态环境中的任务执行，这样可以通过感知来指导认知行为和高层次的决策规划，这种方法可以显著提高任务完成的准确性和效率。

与环境对话：使用大型语言模型进行交互式多模态感知

Chat with the Environment: Interactive Multimodal Perception using Large Language Models

We provide algebraic semantics together with a sound and complete sequent
calculus for information update due to epistemic actions. This semantics is
flexible enough to accommodate incomplete as well as wrong information e.g.due
to secrecy and deceit, as well as nested knowledge. We give a purely algebraic
treatment of the muddy children puzzle, which moreover extends to situations
where the children are allowed to lie and cheat. Epistemic actions, that is,
information exchanges between agents, are modeled as elements of a quantale.
The quantale acts on an underlying right module of epistemic propositions and
facts. The epistemic content is encoded by appearance maps, one pair of (lax)
morphisms for each agent, which preserve the module and quantale structure
respectively. By adjunction, they give rise to epistemic modalities, capturing
the agents' knowledge on propositions and actions. The module action is
epistemic update and gives rise to dynamic modalities - cf. weakest
precondition. This model subsumes the crucial fragment of Baltag, Moss and
Solecki's dynamic epistemic logic, abstracting it in a constructive fashion
while introducing resource-sensitive structure on the epistemic actions.

提供了代数语义及完备的序列演算，以处理由认识行为所引起的信息更新；我们给定一种包容不完整或错误信息（如由于保密或欺骗所致）以及嵌套知识的灵活的语义。