Learning generalizable visual dynamic representation across different
embodied environments is crucial for real-world robotic manipulation. As the
scale and diversity of robot demonstration data are limited, recent works have
turned to large-scale pre-training using human data. However, the morphological
differences between humans and robots introduce a significant human-robot
domain discrepancy, challenging the generalization of these human-data
pre-trained models to downstream manipulation tasks. To address this, we
propose a novel adaptation paradigm that utilizes readily available paired
human-robot video data to bridge the discrepancy. Following this paradigm, our
method exploits a human-robot contrastive alignment loss to align the semantics
of human and robot videos, adapting pre-trained models to the robotic domain in
a parameter-efficient manner. The experiments demonstrate significant
improvements on 25 tasks across three different benchmarks, where the
single-task, language-conditioned multi-task settings are covered, and two
different pre-trained models are evaluated. On the large RLBench benchmark, our
adaptation method achieves an average improvement of $8.9\%$ in success rate
over the pre-trained R3M model across multiple tasks. We will release the code
and models upon acceptance.

利用人机视频数据进行对齐来适应预先训练模型到机器人领域，在多个任务中实现了显著的成功率提升。

缓解机器人操作中的人机领域差异问题

Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for  Robotic Manipulation

Adapting pre-trained neural models to downstream tasks has become the
standard practice for obtaining high-quality models. In this work, we propose a
novel model adaptation paradigm, adapting by pruning, which prunes neural
connections in the pre-trained model to optimise the performance on the target
task; all remaining connections have their weights intact. We formulate
adapting-by-pruning as an optimisation problem with a differentiable loss and
propose an efficient algorithm to prune the model. We prove that the algorithm
is near-optimal under standard assumptions and apply the algorithm to adapt
BERT to some GLUE tasks. Results suggest that our method can prune up to 50%
weights in BERT while yielding similar performance compared to the fine-tuned
full model. We also compare our method with other state-of-the-art pruning
methods and study the topological differences of their obtained sub-networks.

本文提出了一种新的模型适应策略 —— 剪枝适应，将神经模型预先训练的连接进行修剪以优化目标任务的性能，在剩余的连接中保留原有权重；我们将剪枝适应表述为具有可区分损失的优化问题，并提出了一种有效的算法来修剪模型。结果表明，在与微调完整模型相比产生类似性能的情况下，我们的方法可以剪枝 BERT 高达 50％的权重。