We propose an agglomerative Transformer (AGER) that enables Transformer-based
human-object interaction (HOI) detectors to flexibly exploit extra
instance-level cues in a single-stage and end-to-end manner for the first time.
AGER acquires instance tokens by dynamically clustering patch tokens and
aligning cluster centers to instances with textual guidance, thus enjoying two
benefits: 1) Integrality: each instance token is encouraged to contain all
discriminative feature regions of an instance, which demonstrates a significant
improvement in the extraction of different instance-level cues and subsequently
leads to a new state-of-the-art performance of HOI detection with 36.75 mAP on
HICO-Det. 2) Efficiency: the dynamical clustering mechanism allows AGER to
generate instance tokens jointly with the feature learning of the Transformer
encoder, eliminating the need of an additional object detector or instance
decoder in prior methods, thus allowing the extraction of desirable extra cues
for HOI detection in a single-stage and end-to-end pipeline. Concretely, AGER
reduces GFLOPs by 8.5% and improves FPS by 36%, even compared to a vanilla
DETR-like pipeline without extra cue extraction.

我们提出了一种聚合 Transformer (AGER)，首次实现了基于 Transformer 的人 - 物互动 (HOI) 检测器在单阶段、端到端方式中灵活地利用额外的实例级线索。AGER 通过动态聚类与文本引导相结合，获取实例标记，从而在提取不同实例级线索方面呈现出显著改进，HOI 检测在 HICO-Det 上获得了 36.75 的 mAP，达到了最新的性能水平。