Predicting accurate future trajectories of multiple agents is essential for
autonomous systems, but is challenging due to the complex agent interaction and
the uncertainty in each agent's future behavior. Forecasting multi-agent
trajectories requires modeling two key dimensions: (1) time dimension, where we
model the influence of past agent states over future states; (2) social
dimension, where we model how the state of each agent affects others. Most
prior methods model these two dimensions separately, e.g., first using a
temporal model to summarize features over time for each agent independently and
then modeling the interaction of the summarized features with a social model.
This approach is suboptimal since independent feature encoding over either the
time or social dimension can result in a loss of information. Instead, we would
prefer a method that allows an agent's state at one time to directly affect
another agent's state at a future time. To this end, we propose a new
Transformer, AgentFormer, that jointly models the time and social dimensions.
The model leverages a sequence representation of multi-agent trajectories by
flattening trajectory features across time and agents. Since standard attention
operations disregard the agent identity of each element in the sequence,
AgentFormer uses a novel agent-aware attention mechanism that preserves agent
identities by attending to elements of the same agent differently than elements
of other agents. Based on AgentFormer, we propose a stochastic multi-agent
trajectory prediction model that can attend to features of any agent at any
previous timestep when inferring an agent's future position. The latent intent
of all agents is also jointly modeled, allowing the stochasticity in one
agent's behavior to affect other agents. Our method substantially improves the
state of the art on well-established pedestrian and autonomous driving
datasets.

论文提出了一个新的 Transformer 模型 AgentFormer，该模型可以联合模拟时间和社交两个维度，用于预测多智能体行为，同时考虑了多智能体的影响以及不确定性。该模型具有良好的性能，并在公共数据集上取得了 state-of-the-art 的结果。

AgentFormer: 面向多智能体时空预测的智能体感知 Transformer

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent  Forecasting

Inferring interactions from multi-agent trajectories has broad applications
in physics, vision and robotics. Neural relational inference (NRI) is a deep
generative model that can reason about relations in complex dynamics without
supervision. In this paper, we take a careful look at this approach for
relational inference in multi-agent trajectories. First, we discover that NRI
can be fundamentally limited without sufficient long-term observations. Its
ability to accurately infer interactions degrades drastically for short output
sequences. Next, we consider a more general setting of relational inference
when interactions are changing overtime. We propose an extension ofNRI, which
we call the DYnamic multi-AgentRelational Inference (DYARI) model that can
reason about dynamic relations. We conduct exhaustive experiments to study the
effect of model architecture, under-lying dynamics and training scheme on the
performance of dynamic relational inference using a simulated physics system.
We also showcase the usage of our model on real-world multi-agent basketball
trajectories.

本文介绍了神经关系推理 (NRI) 模型在多智能体轨迹的关系推理领域的应用。作者发现没有足够长期的记录会严重损害 NRI 模型的推理结果，为此提出了一种叫做 DYARI 模型的扩展模型，能够处理关系动态变化的情况。作者通过对模型结构、底层动态和训练方案的详细研究，并在模拟物理系统和真实多智能体篮球轨迹上验证了 DYARI 模型的有效性。