Effective interlocutors account for the uncertain goals, beliefs, and
emotions of others. But even the best human conversationalist cannot perfectly
anticipate the trajectory of a dialogue. How well can language models represent
inherent uncertainty in conversations? We propose FortUne Dial, an expansion of
the long-standing "conversation forecasting" task: instead of just accuracy,
evaluation is conducted with uncertainty-aware metrics, effectively enabling
abstention on individual instances. We study two ways in which language models
potentially represent outcome uncertainty (internally, using scores and
directly, using tokens) and propose fine-tuning strategies to improve
calibration of both representations. Experiments on eight difficult negotiation
corpora demonstrate that our proposed fine-tuning strategies (a traditional
supervision strategy and an off-policy reinforcement learning strategy) can
calibrate smaller open-source models to compete with pre-trained models 10x
their size.

有效的对话者必须考虑他人的不确定目标、信念和情绪，但是即使是最好的人类对话者也无法完美地预测对话的轨迹。我们提出了 FortUne Dial，对长期存在的 “对话预测” 任务进行了扩展：不仅评估准确性，还使用了具有不确定性意识的度量标准，有效地使个别实例中可以放弃预测。我们研究了语言模型在代表对话结果不确定性方面的两种方式（内部使用分数，直接使用标记），并提出了改进这两种表示的校准的微调策略。在八个困难的谈判语料库上的实验证明，我们提出的微调策略（传统的监督策略和离线策略增强学习策略）能够使规模较小的开源模型与其体积为 10 倍的预训练模型竞争。