Multi-modal data abounds in biomedicine, such as radiology images and
reports. Interpreting this data at scale is essential for improving clinical
care and accelerating clinical research. Biomedical text with its complex
semantics poses additional challenges in vision--language modelling compared to
the general domain, and previous work has used insufficiently adapted models
that lack domain-specific language understanding. In this paper, we show that
principled textual semantic modelling can substantially improve contrastive
learning in self-supervised vision--language processing. We release a language
model that achieves state-of-the-art results in radiology natural language
inference through its improved vocabulary and novel language pretraining
objective leveraging semantics and discourse characteristics in radiology
reports. Further, we propose a self-supervised joint vision--language approach
with a focus on better text modelling. It establishes new state of the art
results on a wide range of publicly available benchmarks, in part by leveraging
our new domain-specific language model. We release a new dataset with
locally-aligned phrase grounding annotations by radiologists to facilitate the
study of complex semantic modelling in biomedical vision--language processing.
A broad evaluation, including on this new dataset, shows that our contrastive
learning approach, aided by textual-semantic modelling, outperforms prior
methods in segmentation tasks, despite only using a global-alignment objective.

本文介绍了一种基于语义模型的对比学习方法，该方法在生物医学领域的文本 - 语言建模中取得了目前最先进的结果，利用该方法可以更好地理解医学图像和报告。

利用文本语义改进生物医学视觉 - 语言处理

Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing

The ability to accurately represent sentences is central to language
understanding. We describe a convolutional architecture dubbed the Dynamic
Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of
sentences. The network uses Dynamic k-Max Pooling, a global pooling operation
over linear sequences. The network handles input sentences of varying length
and induces a feature graph over the sentence that is capable of explicitly
capturing short and long-range relations. The network does not rely on a parse
tree and is easily applicable to any language. We test the DCNN in four
experiments: small scale binary and multi-class sentiment prediction, six-way
question classification and Twitter sentiment prediction by distant
supervision. The network achieves excellent performance in the first three
tasks and a greater than 25% error reduction in the last task with respect to
the strongest baseline.

本文介绍了一种称为动态卷积神经网络的卷积架构，它使用动态 k-Max 池化进行句子的语义建模，并通过四项实验（二进制和多类情感预测，六路问题分类和推特情感预测）展示了出色的表现。