Medical image interpretation is central to most clinical applications such as
disease diagnosis, treatment planning, and prognostication. In clinical
practice, radiologists examine medical images and manually compile their
findings into reports, which can be a time-consuming process. Automated
approaches to radiology report generation, therefore, can reduce radiologist
workload and improve efficiency in the clinical pathway. While recent
deep-learning approaches for automated report generation from medical images
have seen some success, most studies have relied on image-derived features
alone, ignoring non-imaging patient data. Although a few studies have included
the word-level contexts along with the image, the use of patient demographics
is still unexplored. This paper proposes a novel multi-modal transformer
network that integrates chest x-ray (CXR) images and associated patient
demographic information, to synthesise patient-specific radiology reports. The
proposed network uses a convolutional neural network to extract visual features
from CXRs and a transformer-based encoder-decoder network that combines the
visual features with semantic text embeddings of patient demographic
information, to synthesise full-text radiology reports. Data from two public
databases were used to train and evaluate the proposed approach. CXRs and
reports were extracted from the MIMIC-CXR database and combined with
corresponding patients' data MIMIC-IV. Based on the evaluation metrics used
including patient demographic information was found to improve the quality of
reports generated using the proposed approach, relative to a baseline network
trained using CXRs alone. The proposed approach shows potential for enhancing
radiology report generation by leveraging rich patient metadata and combining
semantic text embeddings derived thereof, with medical image-derived visual
features.

该研究提出了一种新颖的多模态变压器网络，将胸部 X 光片（CXR）图像与相关的患者人口统计信息相结合，合成特定患者的放射学报告，并通过综合得出的语义文本嵌入和医学图像 - derived 视觉特征的变压器编码器 - 解码器网络，以提高报告生成的质量。

使用非影像数据为条件的 Transformers 生成放射学报告

Radiology Report Generation Using Transformers Conditioned with  Non-imaging Data

This paper proposes a novel multi-modal transformer network for detecting
actions in untrimmed videos. To enrich the action features, our transformer
network utilizes a new multi-modal attention mechanism that computes the
correlations between different spatial and motion modalities combinations.
Exploring such correlations for actions has not been attempted previously. To
use the motion and spatial modality more effectively, we suggest an algorithm
that corrects the motion distortion caused by camera movement. Such motion
distortion, common in untrimmed videos, severely reduces the expressive power
of motion features such as optical flow fields. Our proposed algorithm
outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and
ActivityNet. We also conducted comparative experiments on our new instructional
activity dataset, including a large set of challenging classroom videos
captured from elementary schools.

本文提出了一种新的多模态转换器网络，用于检测未修剪视频中的动作，利用新的多模态注意机制计算不同空间和动态模态组合之间的相关性，并提出一种算法来纠正相机运动引起的动态变形，该算法在 THUMOS14 和 ActivityNet 两个公共基准测试上优于现有方法，在我们的新教育活动数据集上进行了比较实验，该数据集包括从小学捕捉的大量具有挑战性的课堂视频。