Text-conditioned human motion generation has experienced significant
advancements with diffusion models trained on extensive motion capture data and
corresponding textual annotations. However, extending such success to 3D
dynamic human-object interaction (HOI) generation faces notable challenges,
primarily due to the lack of large-scale interaction data and comprehensive
descriptions that align with these interactions. This paper takes the
initiative and showcases the potential of generating human-object interactions
without direct training on text-interaction pair data. Our key insight in
achieving this is that interaction semantics and dynamics can be decoupled.
Being unable to learn interaction semantics through supervised training, we
instead leverage pre-trained large models, synergizing knowledge from a large
language model and a text-to-motion model. While such knowledge offers
high-level control over interaction semantics, it cannot grasp the intricacies
of low-level interaction dynamics. To overcome this issue, we further introduce
a world model designed to comprehend simple physics, modeling how human actions
influence object motion. By integrating these components, our novel framework,
InterDreamer, is able to generate text-aligned 3D HOI sequences in a zero-shot
manner. We apply InterDreamer to the BEHAVE and CHAIRS datasets, and our
comprehensive experimental analysis demonstrates its capability to generate
realistic and coherent interaction sequences that seamlessly align with the
text directives.

通过使用预训练的大型模型和文本到动作模型，本文介绍了一种名为 InterDreamer 的框架，能够以零样本的方式生成与文本指令无缝对齐的逼真和连贯的 3D 人物 - 物体交互序列。

InterDreamer: 零激活文本向三维动态人物 - 物体交互

InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

The excellent generalization, contextual learning, and emergence abilities in
the pre-trained large models (PLMs) handle specific tasks without direct
training data, making them the better foundation models in the adversarial
domain adaptation (ADA) methods to transfer knowledge learned from the source
domain to target domains. However, existing ADA methods fail to account for the
confounder properly, which is the root cause of the source data distribution
that differs from the target domains. This study proposes an adversarial domain
adaptation with confounder balancing for PLMs fine-tuning (ADA-CBF). The
ADA-CBF includes a PLM as the foundation model for a feature extractor, a
domain classifier and a confounder classifier, and they are jointly trained
with an adversarial loss. This loss is designed to improve the domain-invariant
representation learning by diluting the discrimination in the domain
classifier. At the same time, the adversarial loss also balances the confounder
distribution among source and unmeasured domains in training. Compared to
existing ADA methods, ADA-CBF can correctly identify confounders in
domain-invariant features, thereby eliminating the confounder biases in the
extracted features from PLMs. The confounder classifier in ADA-CBF is designed
as a plug-and-play and can be applied in the confounder measurable,
unmeasurable, or partially measurable environments. Empirical results on
natural language processing and computer vision downstream tasks show that
ADA-CBF outperforms the newest GPT-4, LLaMA2, ViT and ADA methods.

该研究提出了一种带有混淆量衡的对抗域自适应预训练模型微调方法，该方法在确定领域不变特征的基础上，通过对抗损失来减弱领域分类器中的歧视，并在训练中平衡源域和未测量域之间的混淆分布，从而实现了对从预训练模型提取的特征中的混淆因素进行消除。

预训练大模型微调的对抗领域适应中的混淆平衡

Confounder Balancing in Adversarial Domain Adaptation for Pre-Trained  Large Models Fine-Tuning

The recent upsurge in pre-trained large models (e.g. GPT-4) has swept across
the entire deep learning community. Such powerful large language models (LLMs)
demonstrate advanced generative ability and multimodal understanding
capability, which quickly achieve new state-of-the-art performances on a
variety of benchmarks. The pre-trained LLM usually plays the role as a
universal AI model that can conduct various tasks, including context reasoning,
article analysis and image content comprehension. However, considering the
prohibitively high memory and computational cost for implementing such a large
model, the conventional models (such as CNN and ViT), are still essential for
many visual perception tasks. In this paper, we propose to enhance the
representation ability of ordinary vision models for perception tasks (e.g.
image classification) by taking advantage of large pre-trained models. We
present a new learning paradigm in which the knowledge extracted from large
pre-trained models are utilized to help models like CNN and ViT learn enhanced
representations and achieve better performance. Firstly, we curate a high
quality description set by prompting a multimodal LLM to generate descriptive
text for all training images. Furthermore, we feed these detailed descriptions
into a pre-trained encoder to extract text embeddings with rich semantic
information that encodes the content of images. During training, text
embeddings will serve as extra supervising signals and be aligned with image
representations learned by vision models. The alignment process helps vision
models learn better and achieve higher accuracy with the assistance of
pre-trained LLMs. We conduct extensive experiments to verify that the proposed
algorithm consistently improves the performance for various vision models with
heterogeneous architectures.

本文提出一种新的学习范式，通过利用提前训练的大型模型来增强传统视觉模型的表示能力，并在多个视觉模型中验证了该算法的有效性，可以提高图像分类等任务的准确性。