This study delves into the realm of multi-modality (i.e., video and motion
modalities) human behavior understanding by leveraging the powerful
capabilities of Large Language Models (LLMs). Diverging from recent LLMs
designed for video-only or motion-only understanding, we argue that
understanding human behavior necessitates joint modeling from both videos and
motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics
and semantics effectively. In light of this, we present MotionLLM, a
straightforward yet effective framework for human motion understanding,
captioning, and reasoning. Specifically, MotionLLM adopts a unified
video-motion training strategy that leverages the complementary advantages of
existing coarse video-text data and fine-grained motion-text data to glean rich
spatial-temporal insights. Furthermore, we collect a substantial dataset,
MoVid, comprising diverse videos, motions, captions, and instructions.
Additionally, we propose the MoVid-Bench, with carefully manual annotations,
for better evaluation of human behavior understanding on video and motion.
Extensive experiments show the superiority of MotionLLM in the caption,
spatial-temporal comprehension, and reasoning ability.

该研究通过利用大型语言模型（LLMs）的强大功能，深入探讨多模态（即视频和运动模态）人类行为理解。与最近为仅视频或仅动作理解设计的 LLMs 不同，我们认为理解人类行为需要从视频和运动序列（例如 SMPL 序列）中进行联合建模，以有效捕捉微妙的身体部位动态和语义。因此，我们提出了 MotionLLM，一个简单而有效的人类动作理解、字幕生成和推理框架。具体而言，MotionLLM 采用统一的视频 - 动作训练策略，利用现有的粗糙视频 - 文本数据和细粒度动作 - 文本数据的互补优势，获取丰富的时空洞察力。此外，我们还收集了一个实质性的数据集 MoVid，其中包含了多样的视频、动作、字幕和说明。此外，我们提出了经过精心手动注释的 MoVid-Bench，以更好地评估在视频和动作上对人类行为理解的能力。广泛的实验结果表明 MotionLLM 在字幕生成、时空理解和推理能力方面具有优势。

MotionLLM：从人体动作和视频中理解人类行为

MotionLLM: Understanding Human Behaviors from Human Motions and Videos

Recent advancements in Multimodal Large Language Models (MM-LLMs) have
demonstrated promising potential in terms of generalization and robustness when
applied to different modalities. While previous works have already achieved 3D
human motion generation using various approaches including language modeling,
they mostly % are mostly carefully designed use specialized architecture and
are restricted to single-human motion generation. Inspired by the success of
MM-LLMs, we propose MotionLLM, a simple and general framework that can achieve
single-human, multi-human motion generation, and motion captioning by
fine-tuning pre-trained LLMs. Specifically, we encode and quantize motions into
discrete LLM-understandable tokens, which results in a unified vocabulary
consisting of both motion and text tokens. With only 1--3% parameters of the
LLMs trained by using adapters, our single-human motion generation achieves
comparable results to those diffusion models and other trained-from-scratch
transformer-based models. Additionally, we show that our approach is scalable
and flexible, allowing easy extension to multi-human motion generation through
autoregressive generation of single-human motions. Project page:
this https URL

通过使用预训练的 LLM 模型进行微调，本研究提出了 MotionLLM，一个简单且通用的框架，可实现单人和多人运动生成以及动作字幕生成。