Recent works have shown that large models pretrained on common visual
learning tasks can provide useful representations for a wide range of
specialized perception problems, as well as a variety of robotic manipulation
tasks. While prior work on robotic manipulation has predominantly used frozen
pretrained features, we demonstrate that in robotics this approach can fail to
reach optimal performance, and that fine-tuning of the full model can lead to
significantly better results. Unfortunately, fine-tuning disrupts the
pretrained visual representation, and causes representational drift towards the
fine-tuned task thus leading to a loss of the versatility of the original
model. We introduce "lossless adaptation" to address this shortcoming of
classical fine-tuning. We demonstrate that appropriate placement of our
parameter efficient adapters can significantly reduce the performance gap
between frozen pretrained representations and full end-to-end fine-tuning
without changes to the original representation and thus preserving original
capabilities of the pretrained model. We perform a comprehensive investigation
across three major model architectures (ViTs, NFNets, and ResNets), supervised
(ImageNet-1K classification) and self-supervised pretrained weights (CLIP,
BYOL, Visual MAE) in 3 task domains and 35 individual tasks, and demonstrate
that our claims are strongly validated in various settings.

本文提出 “无损调适” 方案，以解决 fine-tuning 过程对预训练模型通用性的破坏，实验证明此方案在各种任务和模型（ViTs，NFNets 和 ResNets）上表现良好。

预训练视觉模型的无损调整在机器人操作中的应用

Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation

We explore how a general AI algorithm can be used for 3D scene understanding
to reduce the need for training data. More exactly, we propose a modification
of the Monte Carlo Tree Search (MCTS) algorithm to retrieve objects and room
layouts from noisy RGB-D scans. While MCTS was developed as a game-playing
algorithm, we show it can also be used for complex perception problems. Our
adapted MCTS algorithm has few easy-to-tune hyperparameters and can optimise
general losses. We use it to optimise the posterior probability of objects and
room layout hypotheses given the RGB-D data. This results in an
analysis-by-synthesis approach that explores the solution space by rendering
the current solution and comparing it to the RGB-D observations. To perform
this exploration even more efficiently, we propose simple changes to the
standard MCTS' tree construction and exploration policy. We demonstrate our
approach on the ScanNet dataset. Our method often retrieves configurations that
are better than some manual annotations, especially on layouts.

本论文探讨了如何利用通用 AI 算法进行 3D 场景理解来减少训练数据的需求，并提出了一种修改的蒙特卡洛树搜索算法，以从嘈杂的 RGB-D 扫描中检索对象和房间布局。在 ScanNet 数据集上应用本算法可以得到比一些手动注释更好的配置。

3D 场景理解的蒙特卡罗场景搜索

Monte Carlo Scene Search for 3D Scene Understanding

Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require
machine agents to interpret natural language instructions and learn to act in
visually realistic environments to achieve navigation goals. The overall task
requires competence in several perception problems: successful agents combine
spatio-temporal, vision and language understanding to produce appropriate
action sequences. Our approach adapts pre-trained vision and language
representations to relevant in-domain tasks making them more effective for VLN.
Specifically, the representations are adapted to solve both a cross-modal
sequence alignment and sequence coherence task. In the sequence alignment task,
the model determines whether an instruction corresponds to a sequence of visual
frames. In the sequence coherence task, the model determines whether the
perceptual sequences are predictive sequentially in the instruction-conditioned
latent space. By transferring the domain-adapted representations, we improve
competitive agents in R2R as measured by the success rate weighted by path
length (SPL) metric.

我们的方法将预先训练的视觉和语言表示技术适应于相关领域的任务，通过在跨模态序列对齐和序列一致性任务中适应这些表示技术，从而提高了在 Room-to-Room（R2R） 带权路径成功率（SPL）度量方面的性能。