Large-scale pretrained models have proven immensely valuable in handling
data-intensive modalities like text and image. However, fine-tuning these
models for certain specialized modalities, such as protein sequence and cosmic
ray, poses challenges due to the significant modality discrepancy and scarcity
of labeled data. In this paper, we propose an end-to-end method, PaRe, to
enhance cross-modal fine-tuning, aiming to transfer a large-scale pretrained
model to various target modalities. PaRe employs a gating mechanism to select
key patches from both source and target data. Through a modality-agnostic Patch
Replacement scheme, these patches are preserved and combined to construct
data-rich intermediate modalities ranging from easy to hard. By gradually
intermediate modality generation, we can not only effectively bridge the
modality gap to enhance stability and transferability of cross-modal
fine-tuning, but also address the challenge of limited data in the target
modality by leveraging enriched intermediate modality data. Compared with
hand-designed, general-purpose, task-specific, and state-of-the-art cross-modal
fine-tuning approaches, PaRe demonstrates superior performance across three
challenging benchmarks, encompassing more than ten modalities.

我们提出了一种用于增强跨模式微调的端到端方法 PaRe，通过使用选择机制和修补替换方案，逐渐生成丰富的中间模式数据，以有效弥合模态差异并提升跨模式微调的稳定性和可传递性。与手工设计、通用目标、任务特定和最先进的跨模式微调方法相比，PaRe 在三个具有挑战性的基准测试中表现出卓越性能，涵盖了十多个模式。

优化跨模态微调，逐步生成中间模态

Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality  Generation

ORCA (Shen et al., 2023) is a recent technique for cross-modal fine-tuning,
i.e., applying pre-trained transformer models to modalities beyond their
training data. The technique consists primarily of training an embedder and
fine-tuning the embedder and model. Despite its high performance on a variety
of downstream tasks, we do not understand precisely how each of these
components contribute to ORCA's success. Therefore, we run a series of
ablations and find that embedder training does not help 2D tasks at all,
contrary to what the original paper posits. In 1D tasks, some amount of
embedder training is necessary but more is not better. In 4 out of 6 datasets
we experiment with, it is model fine-tuning that makes the biggest difference.
Through our ablations and baselines, we contribute a better understanding of
the individual components of ORCA.

ORCA 是一种最近的跨模态微调技术，通过训练嵌入器和微调嵌入器和模型，我们发现在跨模态微调的成功中，模型微调是最重要的组成部分之一。