In this paper, we explore the impact of adding tactile sensation to video
prediction models for physical robot interactions. Predicting the impact of
robotic actions on the environment is a fundamental challenge in robotics.
Current methods leverage visual and robot action data to generate video
predictions over a given time period, which can then be used to adjust robot
actions. However, humans rely on both visual and tactile feedback to develop
and maintain a mental model of their physical surroundings. In this paper, we
investigate the impact of integrating tactile feedback into video prediction
models for physical robot interactions. We propose three multi-modal
integration approaches and compare the performance of these tactile-enhanced
video prediction models. Additionally, we introduce two new datasets of robot
pushing that use a magnetic-based tactile sensor for unsupervised learning. The
first dataset contains visually identical objects with different physical
properties, while the second dataset mimics existing robot-pushing datasets of
household object clusters. Our results demonstrate that incorporating tactile
feedback into video prediction models improves scene prediction accuracy and
enhances the agent's perception of physical interactions and understanding of
cause-effect relationships during physical robot interactions.

本论文探究将触觉感应应用于物理机器人相互作用的视频预测模型，结果表明增加触觉反馈有助于提高场景预测准确性和增强机器人在物理相互作用过程中的感知和理解能力。

结合视觉和触觉感知进行视频预测

Combining Vision and Tactile Sensation for Video Prediction

Designing video prediction models that account for the inherent uncertainty
of the future is challenging. Most works in the literature are based on
stochastic image-autoregressive recurrent networks, which raises several
performance and applicability issues. An alternative is to use fully latent
temporal models which untie frame synthesis and temporal dynamics. However, no
such model for stochastic video prediction has been proposed in the literature
yet, due to design and training difficulties. In this paper, we overcome these
difficulties by introducing a novel stochastic temporal model whose dynamics
are governed in a latent space by a residual update rule. This first-order
scheme is motivated by discretization schemes of differential equations. It
naturally models video dynamics as it allows our simpler, more interpretable,
latent model to outperform prior state-of-the-art methods on challenging
datasets.

介绍了一种基于残差更新规则的新型随机时间模型，可用于视频预测，并在复杂数据集上胜过先前的最先进方法。

随机潜在残差视频预测

Stochastic Latent Residual Video Prediction

Model-free reinforcement learning (RL) can be used to learn effective
policies for complex tasks, such as Atari games, even from image observations.
However, this typically requires very large amounts of interaction --
substantially more, in fact, than a human would need to learn the same games.
How can people learn so quickly? Part of the answer may be that people can
learn how the game works and predict which actions will lead to desirable
outcomes. In this paper, we explore how video prediction models can similarly
enable agents to solve Atari games with fewer interactions than model-free
methods. We describe Simulated Policy Learning (SimPLe), a complete model-based
deep RL algorithm based on video prediction models and present a comparison of
several model architectures, including a novel architecture that yields the
best results in our setting. Our experiments evaluate SimPLe on a range of
Atari games in low data regime of 100k interactions between the agent and the
environment, which corresponds to two hours of real-time play. In most games
SimPLe outperforms state-of-the-art model-free algorithms, in some games by
over an order of magnitude.

这篇文章介绍了基于视频预测模型的 Simulated Policy Learning 方法，该方法通过在仅与环境交互 100k 次（两小时实时游戏）的情况下，在多个 Atari 游戏中实现比现有的基于模型无关的方法更好的表现。