Prior work on selective prediction minimizes incorrect predictions from
vision-language models (VLMs) by allowing them to abstain from answering when
uncertain. However, when deploying a vision-language system with low tolerance
for inaccurate predictions, selective prediction may be over-cautious and
abstain too frequently, even on many correct predictions. We introduce
ReCoVERR, an inference-time algorithm to reduce the over-abstention of a
selective vision-language system without decreasing prediction accuracy. When
the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries
to find relevant clues in the image that provide additional evidence for the
prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects
high-confidence evidences, and if enough evidence confirms the prediction the
system makes a prediction instead of abstaining. ReCoVERR enables two VLMs,
BLIP2 and InstructBLIP, to answer up to 20% more questions on the A-OKVQA task
than vanilla selective prediction without decreasing system accuracy, thus
improving overall system reliability.

通过推理时间算法减少选择性视觉语言系统的过度式避免并提高系统可靠性。

选择性预测：降低视觉语言推理中的不必要弃权

Selective "Selective Prediction": Reducing Unnecessary Abstention in  Vision-Language Reasoning

Procedural planning, which entails decomposing a high-level goal into a
sequence of temporally ordered steps, is an important yet intricate task for
machines. It involves integrating common-sense knowledge to reason about
complex contextualized situations that are often counterfactual, e.g.
"scheduling a doctor's appointment without a phone". While current approaches
show encouraging results using large language models (LLMs), they are hindered
by drawbacks such as costly API calls and reproducibility issues. In this
paper, we advocate planning using smaller language models. We present PlaSma, a
novel two-pronged approach to endow small language models with procedural
knowledge and (counterfactual) planning capabilities. More concretely, we
develop symbolic procedural knowledge distillation to enhance the implicit
knowledge in small language models and an inference-time algorithm to
facilitate more structured and accurate reasoning. In addition, we introduce a
novel task, Counterfactual Planning, that requires a revision of a plan to cope
with a counterfactual situation. In both the original and counterfactual
setting, we show that orders-of-magnitude smaller models (770M-11B parameters)
can compete and often surpass their larger teacher models' capabilities.

本文提出了 PlaSma，一种利用小型语言模型赋予程序化知识和（反事实的）规划能力的新方法，并介绍了计划的符号化程序知识蒸馏和推理时间算法以及一个新的任务，即对计划进行修订以应对反事实情况的 “反事实规划”，并展示了在原始和反事实情况下，规模比大的教师模型（770M-11B 参数）小几个数量级的模型可以达到甚至超越它们的能力。