Audio visual segmentation (AVS) aims to segment the sounding objects for each
frame of a given video. To distinguish the sounding objects from silent ones,
both audio-visual semantic correspondence and temporal interaction are
required. The previous method applies multi-frame cross-modal attention to
conduct pixel-level interactions between audio features and visual features of
multiple frames simultaneously, which is both redundant and implicit. In this
paper, we propose an Audio-Queried Transformer architecture, AQFormer, where we
define a set of object queries conditioned on audio information and associate
each of them to particular sounding objects. Explicit object-level semantic
correspondence between audio and visual modalities is established by gathering
object information from visual features with predefined audio queries. Besides,
an Audio-Bridged Temporal Interaction module is proposed to exchange sounding
object-relevant information among multiple frames with the bridge of audio
features. Extensive experiments are conducted on two AVS benchmarks to show
that our method achieves state-of-the-art performances, especially 7.1% M_J and
7.6% M_F gains on the MS3 setting.

我们提出了一种基于音频查询的 Transformer 架构 (AQFormer)，通过在视觉特征中利用预定义的音频查询聚集对象信息，建立了音频和视觉模态之间的明确的对象级语义对应关系，并提出了一种基于音频的时间交互模块来在多帧之间交换与声音对象相关的信息，实验结果证明我们的方法在两个 AVS 基准测试集上取得了最先进的性能，尤其在 MS3 设置上取得了 7.1% 的 M_J 增益和 7.6% 的 M_F 增益。

通过音频查询来发现声音对象用于音频视觉分割

Discovering Sounding Objects by Audio Queries for Audio Visual  Segmentation

While it is known that communication facilitates cooperation in multi-agent
settings, it is unclear how to design artificial agents that can learn to
effectively and efficiently communicate with each other. Much research on
communication emergence uses reinforcement learning (RL) and explores
unsituated communication in one-step referential tasks -- the tasks are not
temporally interactive and lack time pressures typically present in natural
communication. In these settings, agents may successfully learn to communicate,
but they do not learn to exchange information concisely -- they tend towards
over-communication and an inefficient encoding. Here, we explore situated
communication in a multi-step task, where the acting agent has to forgo an
environmental action to communicate. Thus, we impose an opportunity cost on
communication and mimic the real-world pressure of passing time. We compare
communication emergence under this pressure against learning to communicate
with a cost on articulation effort, implemented as a per-message penalty (fixed
and progressively increasing). We find that while all tested pressures can
disincentivise over-communication, situated communication does it most
effectively and, unlike the cost on effort, does not negatively impact
emergence. Implementing an opportunity cost on communication in a temporally
extended environment is a step towards embodiment, and might be a pre-condition
for incentivising efficient, human-like communication.

本研究使用强化学习来探索多智能体间的通信，模拟真实世界的压力，并比较实施在多步骤任务中的自然通信和基于非自然通信的良好交流之间的差异，研究结果表明，自然通信可以更有效地激励有效，类似于人类的沟通。