It is a long-lasting goal to design an embodied system that can solve
long-horizon open-world tasks in human-like ways. However, existing approaches
usually struggle with compound difficulties caused by the logic-aware
decomposition and context-aware execution of these tasks. To this end, we
introduce MP5, an open-ended multimodal embodied system built upon the
challenging Minecraft simulator, which can decompose feasible sub-objectives,
design sophisticated situation-aware plans, and perform embodied action
control, with frequent communication with a goal-conditioned active perception
scheme. Specifically, MP5 is developed on top of recent advances in Multimodal
Large Language Models (MLLMs), and the system is modulated into functional
modules that can be scheduled and collaborated to ultimately solve pre-defined
context- and process-dependent tasks. Extensive experiments prove that MP5 can
achieve a 22% success rate on difficult process-dependent tasks and a 91%
success rate on tasks that heavily depend on the context. Moreover, MP5
exhibits a remarkable ability to address many open-ended tasks that are
entirely novel.

设计了一个使用多模态方法的体现系统，基于 Minecraft 仿真器，具备分解子目标、设计情境感知计划、执行体现动作控制以及与目标驱动的主动感知方案频繁通信的能力，在难度较大的任务中获得 22% 的成功率，在严重依赖背景的任务中获得 91% 的成功率，而且表现出在全新的多样任务上具备显著能力。

MP5: 基于主动感知的多模态开放式系统在 Minecraft 中的实体化

MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active  Perception

We propose an embodied system based on the free energy principle (FEP) for
sensorimotor visual perception. We evaluated it in a character-recognition task
using the MNIST dataset. Although the FEP has successfully described a rule
that living things obey mathematically and claims that a biological system
continues to change its internal models and behaviors to minimize the
difference in predicting sensory input, it is not enough to model sensorimotor
visual perception. An embodiment of the system is the key to achieving
sensorimotor visual perception. The proposed embodied system is configured by a
body and memory. The body has an ocular motor system controlling the direction
of eye gaze, which means that the eye can only observe a small focused area of
the environment. The memory is not photographic, but is a generative model
implemented with a variational autoencoder that contains prior knowledge about
the environment, and that knowledge is classified. By limiting body and memory
abilities and operating according to the FEP, the embodied system repeatedly
takes action to obtain the next sensory input based on various potentials of
future sensory inputs. In the evaluation, the inference of the environment was
represented as an approximate posterior distribution of characters (0 - 9). As
the number of repetitions increased, the attention area moved continuously,
gradually reducing the uncertainty of characters. Finally, the probability of
the correct character became the highest among the characters. Changing the
initial attention position provides a different final distribution, suggesting
that the proposed system has a confirmation bias.

提出了一种基于自由能原理的具有实体系统的模型，其包括身体和注意力模型，并以 MNIST 数据集为例刻画了注意力模型的推理过程。