In day-to-day communication, people often approximate the truth - for
example, rounding the time or omitting details - in order to be maximally
helpful to the listener. How do large language models (LLMs) handle such
nuanced trade-offs? To address this question, we use psychological models and
experiments designed to characterize human behavior to analyze LLMs. We test a
range of LLMs and explore how optimization for human preferences or
inference-time reasoning affects these trade-offs. We find that reinforcement
learning from human feedback improves both honesty and helpfulness, while
chain-of-thought prompting skews LLMs towards helpfulness over honesty.
Finally, GPT-4 Turbo demonstrates human-like response patterns including
sensitivity to the conversational framing and listener's decision context. Our
findings reveal the conversational values internalized by LLMs and suggest that
even these abstract values can, to a degree, be steered by zero-shot prompting.

使用心理模型和实验分析了大型语言模型的处理方式，发现强化学习通过人类反馈可提高诚实性和帮助性，而思维链提示则使语言模型偏向帮助性而牺牲了诚实性。最终，GPT-4 Turbo 展现出类似于人类的回应模式，包括对话框架和听者决策背景的敏感性，揭示了语言模型内化的对话价值观，并暗示即使是这些抽象价值也可在无需提示的情况下进行引导。