As we move closer to real-world AI systems, AI agents must be able to deal with multiparty (group) conversations. Recognizing and interpreting multiparty behaviors is challenging, as the system must recognize individual behavioral cues, deal with the complexity of multiple streams of data from multiple people, and recognize the subtle contingent social exchanges that take place amongst group members. To tackle this challenge, we propose the Multiparty-Transformer (Multipar-T), a transformer model for multiparty behavior modeling. The core component of our proposed approach is the Crossperson Attention, which is specifically designed to detect contingent behavior between pairs of people. We verify the effectiveness of Multipar-T on a publicly available video-based group engagement detection benchmark, where it outperforms state-of-the-art approaches in average F-1 scores by 5.2% and individual class F-1 scores by up to 10.0%. Through qualitative analysis, we show that our Crossperson Attention module is able to discover contingent behavior.

为了解决AI系统中多方(群)对话的问题，我们提出了一种名为Multipar-T的变压器模型，其中的Crossperson Attention组件能够检测成对人之间的感应行为，该模型在公共视频小组交互检测基准测试中优于现有技术，并能够以10.0%的F-1分数提高个人分类F-1分数。

Multipar-T: 基于 Transformer 的多方对话捕捉待定行为