Image matching is a fundamental computer vision problem. While learning-based
methods achieve state-of-the-art performance on existing benchmarks, they
generalize poorly to in-the-wild images. Such methods typically need to train
separate models for different scene types and are impractical when the scene
type is unknown in advance. One of the underlying problems is the limited
scalability of existing data construction pipelines, which limits the diversity
of standard image matching datasets. To address this problem, we propose GIM, a
self-training framework for learning a single generalizable model based on any
image matching architecture using internet videos, an abundant and diverse data
source. Given an architecture, GIM first trains it on standard domain-specific
datasets and then combines it with complementary matching methods to create
dense labels on nearby frames of novel videos. These labels are filtered by
robust fitting, and then enhanced by propagating them to distant frames. The
final model is trained on propagated data with strong augmentations. We also
propose ZEB, the first zero-shot evaluation benchmark for image matching. By
mixing data from diverse domains, ZEB can thoroughly assess the cross-domain
generalization performance of different methods. Applying GIM consistently
improves the zero-shot performance of 3 state-of-the-art image matching
architectures; with 50 hours of YouTube videos, the relative zero-shot
performance improves by 8.4%-18.1%. GIM also enables generalization to extreme
cross-domain data such as Bird Eye View (BEV) images of projected 3D point
clouds (Fig. 1(c)). More importantly, our single zero-shot model consistently
outperforms domain-specific baselines when evaluated on downstream tasks
inherent to their respective domains. The video presentation is available at
this https URL

提出了 GIM：一个基于互联网视频的自我训练框架，通过结合标准领域特定数据集和互补匹配方法，在新的视频中创建密集标签，并通过强化增强的传播数据来训练单一通用模型，能够提高跨领域图像匹配的零样本性能。同时，引入了 ZEB 作为图像匹配的首个零样本评估基准。

从互联网视频中学习通用图像匹配器的方法

GIM: Learning Generalizable Image Matcher From Internet Videos

Genome-wide association studies (GWAS) are used to identify relationships
between genetic variations and specific traits. When applied to
high-dimensional medical imaging data, a key step is to extract
lower-dimensional, yet informative representations of the data as traits.
Representation learning for imaging genetics is largely under-explored due to
the unique challenges posed by GWAS in comparison to typical visual
representation learning. In this study, we tackle this problem from the mutual
information (MI) perspective by identifying key limitations of existing
methods. We introduce a trans-modal learning framework Genetic InfoMax (GIM),
including a regularized MI estimator and a novel genetics-informed transformer
to address the specific challenges of GWAS. We evaluate GIM on human brain 3D
MRI data and establish standardized evaluation protocols to compare it to
existing approaches. Our results demonstrate the effectiveness of GIM and a
significantly improved performance on GWAS.

本研究从互信息的角度出发，通过识别现有方法的关键限制，引入了一种跨模态学习框架 Genetic InfoMax (GIM)，包括正则化的互信息估计器和一种新颖的遗传信息变换器，以解决 GWAS 所面临的特定挑战。我们在人类脑 3D MRI 数据上评估 GIM，并建立了标准化评估协议来与现有方法进行比较。我们的结果表明了 GIM 的有效性和在 GWAS 上显著提升的性能。