Videos are more informative than images because they capture the dynamics of
the scene. By representing motion in videos, we can capture dynamic activities.
In this work, we introduce GPT-4 generated motion descriptions that capture
fine-grained motion descriptions of activities and apply them to three action
datasets. We evaluated several video-text models on the task of retrieval of
motion descriptions. We found that they fall far behind human expert
performance on two action datasets, raising the question of whether video-text
models understand motion in videos. To address it, we introduce a method of
improving motion understanding in video-text models by utilizing motion
descriptions. This method proves to be effective on two action datasets for the
motion description retrieval task. The results draw attention to the need for
quality captions involving fine-grained motion information in existing datasets
and demonstrate the effectiveness of the proposed pipeline in understanding
fine-grained motion during video-text retrieval.

通过引入生成的 GPT-4 运动描述应用于三个行动数据集并在运动描述检索任务上评估几种视频 - 文本模型，本研究探讨了视频与图片的信息性差异，着重关注视频 - 文本模型对于视频中运动的理解以及需要在现有数据集中加入细致动作信息的问题，并证明了利用动作描述提高视频 - 文本模型对于细致动作的理解的方法的有效性。

深入了解视频文本模型的运动表示

Diving Deep into the Motion Representation of Video-Text Models

Large Language Models(LLMs) have shown remarkable emergent abilities in
unifying almost all (if not every) NLP tasks. In the human motion-related
realm, however, researchers still develop siloed models for each task. Inspired
by InstuctGPT, and the generalist concept behind Gato, we introduce AvatarGPT,
an All-in-One framework for motion understanding, planning, generations as well
as other tasks such as motion in-between synthesis. AvatarGPT treats each task
as one type of instruction fine-tuned on the shared LLM. All the tasks are
seamlessly interconnected with language as the universal interface,
constituting a closed-loop within the framework. To achieve this, human motion
sequences are first encoded as discrete tokens, which serve as the extended
vocabulary of LLM. Then, an unsupervised pipeline to generate natural language
descriptions of human action sequences from in-the-wild videos is developed.
Finally, all tasks are jointly trained. Extensive experiments show that
AvatarGPT achieves SOTA on low-level tasks, and promising results on high-level
tasks, demonstrating the effectiveness of our proposed All-in-One framework.
Moreover, for the first time, AvatarGPT enables a principled approach by
iterative traversal of the tasks within the closed-loop for unlimited
long-motion synthesis.

AvatarGPT 是一个全能的运动理解、规划、生成等任务的框架，以及基于 LLM 的动作中间合成等其他任务，通过语言作为通用接口，构成一个闭环。通过将人类运动序列编码为离散的标记，并开发一个无监督的管道来从野外视频中生成人类行动序列的自然语言描述，最终实现了各个任务的联合训练。广泛的实验证明，AvatarGPT 在低级任务上达到了最先进水平，在高级任务上取得了有希望的结果，证明了我们提出的全能框架的有效性。此外，AvatarGPT 首次实现了通过闭环内的任务迭代遍历进行无限长运动合成的原则性方法。

AvatarGPT: 综合框架用于动作理解、规划、生成及其他领域

AvatarGPT: All-in-One Framework for Motion Understanding, Planning,  Generation and Beyond

The goal of motion understanding is to establish a reliable mapping between
motion and action semantics, while it is a challenging many-to-many problem. An
abstract action semantic (i.e., walk forwards) could be conveyed by
perceptually diverse motions (walk with arms up or swinging), while a motion
could carry different semantics w.r.t. its context and intention. This makes an
elegant mapping between them difficult. Previous attempts adopted
direct-mapping paradigms with limited reliability. Also, current automatic
metrics fail to provide reliable assessments of the consistency between motions
and action semantics. We identify the source of these problems as the
significant gap between the two modalities. To alleviate this gap, we propose
Kinematic Phrases (KP) that take the objective kinematic facts of human motion
with proper abstraction, interpretability, and generality characteristics.
Based on KP as a mediator, we can unify a motion knowledge base and build a
motion understanding system. Meanwhile, KP can be automatically converted from
motions and to text descriptions with no subjective bias, inspiring Kinematic
Prompt Generation (KPG) as a novel automatic motion generation benchmark. In
extensive experiments, our approach shows superiority over other methods. Our
code and data would be made publicly available at this https URL

运动理解的目标是建立动作语义与运动之间的可靠映射关系，通过使用运动的客观动力学事实作为中介，我们可以统一运动知识库并构建一个运动理解系统，这也激发了运动生成的新型自动基准测试。