Medical image registration is an essential topic in medical image analysis.
In this paper, we propose a method for medical image registration using a
pretrained large language model. We find that using the pretrained large
language model to encode deep features of the medical images in the
registration model can effectively improve image registration accuracy,
indicating the great potential of the large language model in medical image
registration tasks. We use dual encoders to perform deep feature extraction on
image pairs and then input the features into the pretrained large language
model. To adapt the large language model to our registration task, the weights
of the large language model are frozen in the registration model, and an
adapter is utilized to fine-tune the large language model, which aims at (a)
mapping the visual tokens to the language space before the large language model
computing, (b) project the modeled language tokens output from the large
language model to the visual space. Our method combines output features from
the fine-tuned large language model with the features output from each encoder
layer to gradually generate the deformation fields required for registration in
the decoder. To demonstrate the effectiveness of the large prediction model in
registration tasks, we conducted experiments on knee and brain MRI and achieved
state-of-the-art results.

使用预训练的大型语言模型提取医学图像的深度特征来改善图像的注册准确性，实验证明了大型语言模型在医学图像注册任务中的巨大潜力。

LLaMA-Reg: 使用 LLaMA 2 进行非监督式医学图像配准

LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registration

This paper explores a better prediction target for BERT pre-training of
vision transformers. We observe that current prediction targets disagree with
human perception judgment.This contradiction motivates us to learn a perceptual
prediction target. We argue that perceptually similar images should stay close
to each other in the prediction target space. We surprisingly find one simple
yet effective idea: enforcing perceptual similarity during the dVAE training.
Moreover, we adopt a self-supervised transformer model for deep feature
extraction and show that it works well for calculating perceptual similarity.We
demonstrate that such learned visual tokens indeed exhibit better semantic
meanings, and help pre-training achieve superior transfer performance in
various downstream tasks. For example, we achieve $\textbf{84.5\%}$ Top-1
accuracy on ImageNet-1K with ViT-B backbone, outperforming the competitive
method BEiT by $\textbf{+1.3\%}$ under the same pre-training epochs. Our
approach also gets significant improvement on object detection and segmentation
on COCO and semantic segmentation on ADE20K. Equipped with a larger backbone
ViT-H, we achieve the state-of-the-art ImageNet accuracy (\textbf{88.3\%})
among methods using only ImageNet-1K data.

本文探讨了 BERT 预训练视觉变换器的更好预测目标，提出了学习感知预测目标的想法，并在 dVAE 训练过程中实现感知相似性的强制，使用自监督变换模型进行深度特征提取，最终得到学习到更好的视觉 token，取得了在多个任务上优于 BEiT 的性能表现。