This paper studies automated categorization of age-related macular
degeneration (AMD) given a multi-modal input, which consists of a color fundus
image and an optical coherence tomography (OCT) image from a specific eye.
Previous work uses a traditional method, comprised of feature extraction and
classifier training that cannot be optimized jointly. By contrast, we propose a
two-stream convolutional neural network (CNN) that is end-to-end. The CNN's
fusion layer is tailored to the need of fusing information from the fundus and
OCT streams. For generating more multi-modal training instances, we introduce
Loose Pair training, where a fundus image and an OCT image are paired based on
class labels rather than eyes. Moreover, for a visual interpretation of how the
individual modalities make contributions, we extend the class activation
mapping technique to the multi-modal scenario. Experiments on a real-world
dataset collected from an outpatient clinic justify the viability of our
proposal for multi-modal AMD categorization.

本研究提出了一种基于多模态输入的卷积神经网络自动分类年龄相关性黄斑变性的方法，并通过 Loose Pair 训练和类激活映射技术进行了优化，实验证明了该方法在多模态 AMD 分类方面的可行性。