Deep reinforcement learning (DRL) provides a promising way for intelligent
agents (e.g., autonomous vehicles) to learn to navigate complex scenarios.
However, DRL with neural networks as function approximators is typically
considered a black box with little explainability and often suffers from
suboptimal performance, especially for autonomous navigation in highly
interactive multi-agent environments. To address these issues, we propose three
auxiliary tasks with spatio-temporal relational reasoning and integrate them
into the standard DRL framework, which improves the decision making performance
and provides explainable intermediate indicators. We propose to explicitly
infer the internal states (i.e., traits and intentions) of surrounding agents
(e.g., human drivers) as well as to predict their future trajectories in the
situations with and without the ego agent through counterfactual reasoning.
These auxiliary tasks provide additional supervision signals to infer the
behavior patterns of other interactive agents. Multiple variants of framework
integration strategies are compared. We also employ a spatio-temporal graph
neural network to encode relations between dynamic entities, which enhances
both internal state inference and decision making of the ego agent. Moreover,
we propose an interactivity estimation mechanism based on the difference
between predicted trajectories in these two situations, which indicates the
degree of influence of the ego agent on other agents. To validate the proposed
method, we design an intersection driving simulator based on the Intelligent
Intersection Driver Model (IIDM) that simulates vehicles and pedestrians. Our
approach achieves robust and state-of-the-art performance in terms of standard
evaluation metrics and provides explainable intermediate indicators (i.e.,
internal states, and interactivity scores) for decision making.

通过引入空间 - 时间相关推理等辅助任务，将深度强化学习框架与神经网络结合，以提高智能代理（如自动驾驶车辆）在复杂场景中的决策性能和可解释性。首次通过推理和预测他人行为，确定他们对自身行为的影响程度，提供了详尽的中间指标用于决策制定。