Simulation is a crucial component of any robotic system. In order to simulate
correctly, we need to write complex rules of the environment: how dynamic
agents behave, and how the actions of each of the agents affect the behavior of
others. In this paper, we aim to learn a simulator by simply watching an agent
interact with an environment. We focus on graphics games as a proxy of the real
environment. We introduce GameGAN, a generative model that learns to visually
imitate a desired game by ingesting screenplay and keyboard actions during
training. Given a key pressed by the agent, GameGAN "renders" the next screen
using a carefully designed generative adversarial network. Our approach offers
key advantages over existing work: we design a memory module that builds an
internal map of the environment, allowing for the agent to return to previously
visited locations with high visual consistency. In addition, GameGAN is able to
disentangle static and dynamic components within an image making the behavior
of the model more interpretable, and relevant for downstream tasks that require
explicit reasoning over dynamic elements. This enables many interesting
applications such as swapping different components of the game to build new
games that do not exist.

本文介绍了 GameGAN 这个生成模型，它通过学习代理与环境交互中的表现来模拟图形游戏环境，并通过内部环境地图和动静分离的方式实现更高的可解释性和更广泛的应用。