In high-stakes domains like medical reasoning, AI assistants powered by large
language models (LLMs) are yet to be reliable and safe. We identify a key
obstacle towards reliability: existing LLMs are trained to answer any question,
even with incomplete context in the prompt or insufficient parametric
knowledge. We propose to change this paradigm to develop more careful LLMs that
ask follow-up questions to gather necessary and sufficient information and
respond reliably. We introduce MEDIQ, a framework to simulate realistic
clinical interactions, which incorporates a Patient System and an adaptive
Expert System. The Patient may provide incomplete information in the beginning;
the Expert refrains from making diagnostic decisions when unconfident, and
instead elicits missing details from the Patient via follow-up questions. To
evaluate MEDIQ, we convert MEDQA and CRAFT-MD -- medical benchmarks for
diagnostic question answering -- into an interactive setup. We develop a
reliable Patient system and prototype several Expert systems, first showing
that directly prompting state-of-the-art LLMs to ask questions degrades the
quality of clinical reasoning, indicating that adapting LLMs to interactive
information-seeking settings is nontrivial. We then augment the Expert with a
novel abstention module to better estimate model confidence and decide whether
to ask more questions, thereby improving diagnostic accuracy by 20.3%; however,
performance still lags compared to an (unrealistic in practice) upper bound
when full information is given upfront. Further analyses reveal that
interactive performance can be improved by filtering irrelevant contexts and
reformatting conversations. Overall, our paper introduces a novel problem
towards LLM reliability, a novel MEDIQ framework, and highlights important
future directions to extend the information-seeking abilities of LLM assistants
in critical domains.

在高风险领域如医疗推理中，目前大语言模型的 AI 助理尚缺乏可靠性和安全性。本文提出了挑战可靠性的主要障碍：现有的大语言模型被训练以回答任何问题，即使在提问时存在不完整的上下文或不足的参数知识。我们建议通过开发更加谨慎的大语言模型来改变这种模式，让它们能够通过追问问题来收集必要和充分的信息并提供可靠的回答。我们引入了 MEDIQ 框架，用于模拟逼真的临床交互过程，其中包括患者系统和自适应专家系统。患者系统可能在开始阶段提供不完整的信息；当专家系统对决策缺乏信心时，会通过追问问题来获取患者缺失的详细信息，而不会做出诊断决策。为了评估 MEDIQ，我们将 MEDQA 和 CRAFT-MD（用于诊断问题回答的医学基准测试）转化为交互式设置。我们开发了一个可靠的患者系统，并原型化了几个专家系统，首先表明直接命令最先进的大语言模型提问会降低临床推理的质量，表明将大语言模型适应于交互式信息寻求环境是困难的。然后，我们使用新的弃权模型来改进专家系统，更好地估计模型置信度，并决定是否追问更多问题，从而将诊断准确率提高了 20.3%；然而，性能仍然落后于完整信息均提供的理论上限（在实践中不切实际）。进一步分析表明，通过过滤不相关的上下文和重新格式化对话，可以提高交互性能。总体而言，我们的论文介绍了一个关于大语言模型可靠性的新问题，一个新的 MEDIQ 框架，并强调了在关键领域扩展大语言模型助理的信息寻求能力的重要未来方向。

MEDIQ：用于自适应可靠医学推理的问答 LLMs

MEDIQ: Question-Asking LLMs for Adaptive and Reliable Medical Reasoning

Large Language Model (LLM) assistants, such as ChatGPT, have emerged as
potential alternatives to search methods for helping users navigate complex,
feature-rich software. LLMs use vast training data from domain-specific texts,
software manuals, and code repositories to mimic human-like interactions,
offering tailored assistance, including step-by-step instructions. In this
work, we investigated LLM-generated software guidance through a within-subject
experiment with 16 participants and follow-up interviews. We compared a
baseline LLM assistant with an LLM optimized for particular software contexts,
SoftAIBot, which also offered guidelines for constructing appropriate prompts.
We assessed task completion, perceived accuracy, relevance, and trust.
Surprisingly, although SoftAIBot outperformed the baseline LLM, our results
revealed no significant difference in LLM usage and user perceptions with or
without prompt guidelines and the integration of domain context. Most users
struggled to understand how the prompt's text related to the LLM's responses
and often followed the LLM's suggestions verbatim, even if they were incorrect.
This resulted in difficulties when using the LLM's advice for software tasks,
leading to low task completion rates. Our detailed analysis also revealed that
users remained unaware of inaccuracies in the LLM's responses, indicating a gap
between their lack of software expertise and their ability to evaluate the
LLM's assistance. With the growing push for designing domain-specific LLM
assistants, we emphasize the importance of incorporating explainable,
context-aware cues into LLMs to help users understand prompt-based
interactions, identify biases, and maximize the utility of LLM assistants.

大型语言模型（LLM）助手是帮助用户浏览复杂多功能软件的潜在替代方法。我们通过与 16 名参与者的实验和后续访谈对 LLM 生成的软件指导进行了研究。我们比较了基准 LLM 助手与一种针对特定软件背景优化过的 LLM 助手 SoftAIBot，后者还提供了构建适当提示的指南。但令人惊讶的是，虽然 SoftAIBot 优于基准 LLM，但我们的结果显示，无论是否使用提示指南和领域上下文的集成，LLM 的使用和用户感知没有显著差异。大多数用户难以理解提示文本与 LLM 的回应之间的关联，并且通常逐字逐句地遵循 LLM 的建议，即使是错误的。这导致在使用 LLM 的软件任务建议时出现困难，降低了任务完成率。我们的详细分析还表明，用户对 LLM 的回应中的错误毫不知情，这表明他们在软件专业知识的缺乏和评估 LLM 助手的能力之间存在差距。随着设计领域特定 LLM 助手的推动日益增加，我们强调将可解释的、上下文感知的提示融入 LLM 中的重要性，以帮助用户理解基于提示的交互，识别偏见，并最大化 LLM 助手的效用。

基于 LLM 助手何时出错：对基于提示的软件求助交互有效性的研究

Why and When LLM-Based Assistants Can Go Wrong: Investigating the  Effectiveness of Prompt-Based Interactions for Software Help-Seeking

Today, users ask Large language models (LLMs) as assistants to answer queries
that require external knowledge; they ask about the weather in a specific city,
about stock prices, and even about where specific locations are within their
neighborhood. These queries require the LLM to produce code that invokes
external APIs to answer the user's question, yet LLMs rarely produce correct
code on the first try, requiring iterative code refinement upon execution
results. In addition, using LLM assistants to support high query volumes can be
expensive. In this work, we contribute a framework, EcoAssistant, that enables
LLMs to answer code-driven queries more affordably and accurately. EcoAssistant
contains three components. First, it allows the LLM assistants to converse with
an automatic code executor to iteratively refine code or to produce answers
based on the execution results. Second, we use a hierarchy of LLM assistants,
which attempts to answer the query with weaker, cheaper LLMs before backing off
to stronger, expensive ones. Third, we retrieve solutions from past successful
queries as in-context demonstrations to help subsequent queries. Empirically,
we show that EcoAssistant offers distinct advantages for affordability and
accuracy, surpassing GPT-4 by 10 points of success rate with less than 50% of
GPT-4's cost.

大型语言模型（LLMs）能够作为助手回答需要外部知识的查询，通过 EcoAssistant 框架，我们能够更经济、准确地利用 LLM 助手回答以代码为驱动的查询。