Learned communication between agents is a powerful tool when approaching decision-making problems that are hard to overcome by any single agent in isolation. However, continual coordination and communication learning between machine agents or human-machine partnerships remains a challenging open problem. As a stepping stone toward solving the continual communication learning problem, in this paper we contribute a multi-faceted study into what we term Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent with different perceptual access to their shared environment. We seek to establish how different temporal processes and representational choices impact Pavlovian signalling between learning agents. To do so, we introduce a partially observable decision-making domain we call the Frost Hollow. In this domain a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that seeks to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: 1) machine prediction and control learning in a linear walk, and 2) a prediction learning machine interacting with a human participant in a virtual reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning. Our results therefore point to an actionable, constructivist path towards continual communication learning between reinforcement learning agents, with potential impact in a range of real-world settings.

本文针对多智能体或人机伙伴关系中的信号学习问题，探讨了称为帕夫洛夫信号的学习过程，该过程通过一个智能体的学习，为另一个不同感性环境的智能体的决策提供预测。作者通过实验结果表明了帕夫洛夫信号学习的速度、不同时间间隔表示及时序混淆对智能体之间的协调影响，并建立了帕夫洛夫信号学习作为固定信号范式和全自适应通信学习之间的自然桥梁。结果为实现多智能体之间的持续通信学习提供了具体可行的路径。

霜之谷实验：巴甫洛夫信号作为实现代理间协调和通讯的途径