Medical Visual Question Answering (Medical-VQA) aims to to answer clinical
questions regarding radiology images, assisting doctors with decision-making
options. Nevertheless, current Medical-VQA models learn cross-modal
representations through residing vision and texture encoders in dual separate
spaces, which lead to indirect semantic alignment. In this paper, we propose
UnICLAM, a Unified and Interpretable Medical-VQA model through Contrastive
Representation Learning with Adversarial Masking. Specifically, to learn an
aligned image-text representation, we first establish a unified dual-stream
pre-training structure with the gradually soft-parameter sharing strategy.
Technically, the proposed strategy learns a constraint for the vision and
texture encoders to be close in a same space, which is gradually loosened as
the higher number of layers. Moreover, for grasping the unified semantic
representation, we extend the adversarial masking data augmentation to the
contrastive representation learning of vision and text in a unified manner.
Concretely, while the encoder training minimizes the distance between original
and masking samples, the adversarial masking module keeps adversarial learning
to conversely maximize the distance. Furthermore, we also intuitively take a
further exploration to the unified adversarial masking augmentation model,
which improves the potential ante-hoc interpretability with remarkable
performance and efficiency. Experimental results on VQA-RAD and SLAKE public
benchmarks demonstrate that UnICLAM outperforms existing 11 state-of-the-art
Medical-VQA models. More importantly, we make an additional discussion about
the performance of UnICLAM in diagnosing heart failure, verifying that UnICLAM
exhibits superior few-shot adaption performance in practical disease diagnosis.

本篇论文提出了 UnICLAM，一种通过对比表示学习与敌对遮盖进行统一解释的医学视觉问答模型，可用于心力衰竭等疾病的诊断，并且在公共基准上胜过了 11 种最先进的医学视觉问答模型。

UnICLAM: 对抗掩蔽的对比表示学习，实现统一和可解释的医学视觉问答

UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering

Medical datasets often face the problem of data scarcity, as ground truth
labels must be generated by medical professionals. One mitigation strategy is
to pretrain deep learning models on large, unlabelled datasets with
self-supervised learning (SSL). Data augmentations are essential for improving
the generalizability of SSL-trained models, but they are typically handcrafted
and tuned manually. We use an adversarial model to generate masks as
augmentations for 12-lead electrocardiogram (ECG) data, where masks learn to
occlude diagnostically-relevant regions of the ECGs. Compared to random
augmentations, adversarial masking reaches better accuracy when transferring to
to two diverse downstream objectives: arrhythmia classification and gender
classification. Compared to a state-of-art ECG augmentation method 3KG,
adversarial masking performs better in data-scarce regimes, demonstrating the
generalizability of our model.

本研究提出了一种采用敌对掩蔽技术增强医学数据集的自监督学习方法，该方法针对 12 导联心电图数据生成掩蔽增强，相比于传统增强方法在少样本数据集上表现更优，证明了该方法的泛化性。

敌对掩蔽预训练心电图数据提高模型泛化能力用于数据稀缺任务

Pretraining ECG Data with Adversarial Masking Improves Model Generalizability for Data-Scarce Tasks

Recent work has shown the importance of adaptation of broad-coverage
contextualised embedding models on the domain of the target task of interest.
Current self-supervised adaptation methods are simplistic, as the training
signal comes from a small percentage of \emph{randomly} masked-out tokens. In
this paper, we show that careful masking strategies can bridge the knowledge
gap of masked language models (MLMs) about the domains more effectively by
allocating self-supervision where it is needed. Furthermore, we propose an
effective training strategy by adversarially masking out those tokens which are
harder to reconstruct by the underlying MLM. The adversarial objective leads to
a challenging combinatorial optimisation problem over \emph{subsets} of tokens,
which we tackle efficiently through relaxation to a variational lowerbound and
dynamic programming. On six unsupervised domain adaptation tasks involving
named entity recognition, our method strongly outperforms the random masking
strategy and achieves up to +1.64 F1 score improvements.

本文探讨了如何利用适当的遮罩策略来提高基于掩蔽的语言模型在领域自适应任务上的性能，并提出了一种有效的训练策略，即通过对更难以恢复的标记进行对抗性遮蔽，以使模型更好地适应目标任务。通过六个无监督领域适应任务，该方法在命名实体识别方面显著优于随机遮蔽策略，提高了 1.64 个 F1 分数。