In this paper, we propose Multi-View Dreaming, a novel reinforcement learning
agent for integrated recognition and control from multi-view observations by
extending Dreaming. Most current reinforcement learning method assumes a
single-view observation space, and this imposes limitations on the observed
data, such as lack of spatial information and occlusions. This makes obtaining
ideal observational information from the environment difficult and is a
bottleneck for real-world robotics applications. In this paper, we use
contrastive learning to train a shared latent space between different
viewpoints, and show how the Products of Experts approach can be used to
integrate and control the probability distributions of latent states for
multiple viewpoints. We also propose Multi-View DreamingV2, a variant of
Multi-View Dreaming that uses a categorical distribution to model the latent
state instead of the Gaussian distribution. Experiments show that the proposed
method outperforms simple extensions of existing methods in a realistic robot
control task.

本文提出了一个新的强化学习代理，Multi-View Dreaming，用于从多视角观测中进行综合识别和控制。我们使用对比学习来训练不同视角之间的共享潜在空间，并展示了如何使用专家产品方法来集成和控制多个视角的概率分布的潜在状态。我们还提出了 Multi-View DreamingV2，它使用分类分布来建模潜在状态，并证明该方法在实际机器人控制任务中优于现有方法的简单扩展。