Direct image-to-graph transformation is a challenging task that solves object
detection and relationship prediction in a single model. Due to the complexity
of this task, large training datasets are rare in many domains, which makes the
training of large networks challenging. This data sparsity necessitates the
establishment of pre-training strategies akin to the state-of-the-art in
computer vision. In this work, we introduce a set of methods enabling
cross-domain and cross-dimension transfer learning for image-to-graph
transformers. We propose (1) a regularized edge sampling loss for sampling the
optimal number of object relationships (edges) across domains, (2) a domain
adaptation framework for image-to-graph transformers that aligns features from
different domains, and (3) a simple projection function that allows us to
pretrain 3D transformers on 2D input data. We demonstrate our method's utility
in cross-domain and cross-dimension experiments, where we pretrain our models
on 2D satellite images before applying them to vastly different target domains
in 2D and 3D. Our method consistently outperforms a series of baselines on
challenging benchmarks, such as retinal or whole-brain vessel graph extraction.

直接从图像到图表的转换是一项具有挑战性的任务，这一任务要在单个模型中解决对象检测和关系预测。本文提出了一套方法，使得图像到图表转换器能够进行跨领域和跨维度的迁移学习，并通过将模型在 2D 卫星图像上进行预训练，应用于大不相同的 2D 和 3D 目标领域，对具有挑战性的基准进行了验证，如视网膜或整个脑血管图提取，在多个基准测试中方法始终优于一系列基线模型。