We study improving social conversational agents by learning from natural
dialogue between users and a deployed model, without extra annotations. To
implicitly measure the quality of a machine-generated utterance, we leverage
signals like user response length, sentiment and reaction of the future human
utterances in the collected dialogue episodes. Our experiments use the publicly
released deployment data from BlenderBot (Xu et al., 2023). Human evaluation
indicates improvements in our new models over baseline responses; however, we
find that some proxy signals can lead to more generations with undesirable
properties as well. For example, optimizing for conversation length can lead to
more controversial or unfriendly generations compared to the baseline, whereas
optimizing for positive sentiment or reaction can decrease these behaviors.

通过学习用户与已部署模型之间的自然对话，我们研究了提高社交对话代理的方法，无需额外的注释。利用用户响应长度、情感以及未来人类发言中的反应等信号来隐式衡量机器生成发言的质量。我们的实验使用了 BlenderBot（Xu 等，2023 年）的公开发布的部署数据。人工评估表明，我们的新模型在基线回复上有所改进；然而，我们发现一些代理信号也可能导致具有不良特性的生成。例如，优化对话长度可能导致比基线更具争议性或不友好的生成，而优化积极情感或反应则可能减少这些行为。