Visual Question Answering (VQA) has emerged as one of the most challenging
tasks in artificial intelligence due to its multi-modal nature. However, most
existing VQA methods are incapable of handling Knowledge-based Visual Question
Answering (KB-VQA), which requires external knowledge beyond visible contents
to answer questions about a given image. To address this issue, we propose a
novel framework that endows the model with capabilities of answering more
general questions, and achieves a better exploitation of external knowledge
through generating Multiple Clues for Reasoning with Memory Neural Networks
(MCR-MemNN). Specifically, a well-defined detector is adopted to predict
image-question related relation phrases, each of which delivers two
complementary clues to retrieve the supporting facts from external knowledge
base (KB), which are further encoded into a continuous embedding space using a
content-addressable memory. Afterwards, mutual interactions between
visual-semantic representation and the supporting facts stored in memory are
captured to distill the most relevant information in three modalities (i.e.,
image, question, and KB). Finally, the optimal answer is predicted by choosing
the supporting fact with the highest score. We conduct extensive experiments on
two widely-used benchmarks. The experimental results well justify the
effectiveness of MCR-MemNN, as well as its superiority over other KB-VQA
methods.

通过生成多个线索进行推理的记忆神经网络 (MCR-MemNN) 作为一种全新的框架用于视觉问答，可以更好地利用外部知识来回答更一般性的问题，并且在各模态（图像、问题和知识库）中捕捉到最相关的信息。

基于记忆增强的多线索推理的知识驱动视觉问答

Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual  Question Answering

Diagnosis of a clinical condition is a challenging task, which often requires
significant medical investigation. Previous work related to diagnostic
inferencing problems mostly consider multivariate observational data (e.g.
physiological signals, lab tests etc.). In contrast, we explore the problem
using free-text medical notes recorded in an electronic health record (EHR).
Complex tasks like these can benefit from structured knowledge bases, but those
are not scalable. We instead exploit raw text from Wikipedia as a knowledge
source. Memory networks have been demonstrated to be effective in tasks which
require comprehension of free-form text. They use the final iteration of the
learned representation to predict probable classes. We introduce condensed
memory neural networks (C-MemNNs), a novel model with iterative condensation of
memory representations that preserves the hierarchy of features in the memory.
Experiments on the MIMIC-III dataset show that the proposed model outperforms
other variants of memory networks to predict the most probable diagnoses given
a complex clinical scenario.

通过利用维基百科作为知识源，提出了一种使用电子健康记录中记录的自由文本医疗笔记进行医学诊断的新模型 —— 紧凑记忆神经网络（C-MemNNs）。在 MIMIC-III 数据集上的实验证明，该模型是预测复杂临床场景下最可能诊断的最佳选择之一。