Frozen models trained to mimic static datasets can never improve their
performance. Models that can employ internet-retrieval for up-to-date
information and obtain feedback from humans during deployment provide the
promise of both adapting to new information, and improving their performance.
In this work we study how to improve internet-driven conversational skills in
such a learning framework. We collect deployment data, which we make publicly
available, of human interactions, and collect various types of human feedback
-- including binary quality measurements, free-form text feedback, and
fine-grained reasons for failure. We then study various algorithms for
improving from such feedback, including standard supervised learning, rejection
sampling, model-guiding and reward-based learning, in order to make
recommendations on which type of feedback and algorithms work best. We find the
recently introduced Director model (Arora et al., '22) shows significant
improvements over other existing approaches.

研究利用互联网检索获取更新信息并从人类反馈中提升网络聊天技能的算法，并表明了利用 Director 模型可以显著提高性能。