Multimodal machine learning with missing modalities is an increasingly
relevant challenge arising in various applications such as healthcare. This
paper extends the current research into missing modalities to the low-data
regime, i.e., a downstream task has both missing modalities and limited sample
size issues. This problem setting is particularly challenging and also
practical as it is often expensive to get full-modality data and sufficient
annotated training samples. We propose to use retrieval-augmented in-context
learning to address these two crucial issues by unleashing the potential of a
transformer's in-context learning ability. Diverging from existing methods,
which primarily belong to the parametric paradigm and often require sufficient
training samples, our work exploits the value of the available full-modality
data, offering a novel perspective on resolving the challenge. The proposed
data-dependent framework exhibits a higher degree of sample efficiency and is
empirically demonstrated to enhance the classification model's performance on
both full- and missing-modality data in the low-data regime across various
multimodal learning tasks. When only 1% of the training data are available, our
proposed method demonstrates an average improvement of 6.1% over a recent
strong baseline across various datasets and missing states. Notably, our method
also reduces the performance gap between full-modality and missing-modality
data compared with the baseline.

通过使用检索增强上下文学习来解决多模态机器学习中缺失模态和样本有限的问题，提高分类模型性能和样本效率。

借宝于邻：针对多模态学习的上下文学习在缺失模态和数据稀缺情况下的应用

Borrowing Treasures from Neighbors: In-Context Learning for Multimodal  Learning with Missing Modalities and Data Scarcity

With the widespread adoption of large language models (LLMs) in numerous
applications, the challenge of factuality and the propensity for hallucinations
raises significant concerns. To address this issue, particularly in
retrieval-augmented in-context learning, we introduce the hierarchical graph of
thoughts (HGOT), a structured, multi-layered graph approach designed to enhance
the retrieval of pertinent passages during in-context learning. The framework
utilizes the emergent planning capabilities of LLMs, employing the
divide-and-conquer strategy to break down complex queries into manageable
sub-queries. It refines self-consistency majority voting for answer selection,
which incorporates the recently proposed citation recall and precision metrics
to assess the quality of thoughts, linking an answer's credibility
intrinsically to the thought's quality. This methodology introduces a weighted
system in majority voting, prioritizing answers based on the citation quality
of their thoughts. Additionally, we propose a scoring mechanism for evaluating
retrieved passages, considering factors such as citation frequency and quality,
self-consistency confidence, and the retrieval module's ranking. Experiments
reveal that HGOT outperforms other retrieval-augmented in-context learning
methods, including Demonstrate-Search-Predict (DSP), ReAct, Self-Ask, and
Retrieve-then-Read on different datasets by as much as $7\%$, demonstrating its
efficacy in enhancing the factuality of LLMs.

利用大语言模型（LLMs）在检索增强的上下文学习中的层次图思路（HGOT），该框架通过将复杂查询细分为可管理的子查询，并使用最近提出的引文召回和精度指标来评估思想的质量，从而增强了 LLMs 的真实性。