Existing action recognition methods are typically actor-specific due to the
intrinsic topological and apparent differences among the actors. This requires
actor-specific pose estimation (e.g., humans vs. animals), leading to
cumbersome model design complexity and high maintenance costs. Moreover, they
often focus on learning the visual modality alone and single-label
classification whilst neglecting other available information sources (e.g.,
class name text) and the concurrent occurrence of multiple actions. To overcome
these limitations, we propose a new approach called 'actor-agnostic multi-modal
multi-label action recognition,' which offers a unified solution for various
types of actors, including humans and animals. We further formulate a novel
Multi-modal Semantic Query Network (MSQNet) model in a transformer-based object
detection framework (e.g., DETR), characterized by leveraging visual and
textual modalities to represent the action classes better. The elimination of
actor-specific model designs is a key advantage, as it removes the need for
actor pose estimation altogether. Extensive experiments on five publicly
available benchmarks show that our MSQNet consistently outperforms the prior
arts of actor-specific alternatives on human and animal single- and multi-label
action recognition tasks by up to 50%. Code will be released at
this https URL

现有的行动识别方法通常是针对特定的演员，这是由于演员之间的内在拓扑和显而易见的差异造成的。我们提出了一种称为 ' 演员不可知的多模多标签行动识别 ' 的新方法，以统一解决包括人类和动物在内的各种演员类型的问题。我们进一步在一种基于 Transformer 的目标检测框架（如 DETR）中制定了一种新颖的多模态语义查询网络（MSQNet）模型，它利用视觉和文本模态更好地表示行动类别。通过对五个公开可用的基准测试进行广泛的实验证明，我们的 MSQNet 在人类和动物的单标签和多标签行动识别任务上始终优于特定演员的先前方案，提升幅度高达 50%。代码将在此 URL 发布。