Human guidance is often desired in reinforcement learning to improve the
performance of the learning agent. However, human insights are often mere
opinions and educated guesses rather than well-formulated arguments. While
opinions are subject to uncertainty, e.g., due to partial informedness or
ignorance about a problem, they also emerge earlier than hard evidence could be
produced. Thus, guiding reinforcement learning agents through opinions offers
the potential for more performant learning processes, but comes with the
challenge of modeling and managing opinions in a formal way. In this article,
we present a method to guide reinforcement learning agents through opinions. To
this end, we provide an end-to-end method to model and manage advisors'
opinions. To assess the utility of the approach, we evaluate it with synthetic
and human advisors, at different levels of uncertainty, and under multiple
advise strategies. Our results indicate that opinions, even if uncertain,
improve the performance of reinforcement learning agents, resulting in higher
rewards, more efficient exploration, and a better reinforced policy. Although
we demonstrate our approach in a simplified topological running example, our
approach is applicable to complex problems with higher dimensions as well.

人类导引在强化学习中经常被用来提升学习代理的性能。然而，人类的见解通常只是意见和猜测，而不是明确的论证。尽管意见存在不确定性，但它们往往比硬证据早出现。因此，通过意见来引导强化学习代理提供了更有效的学习过程的潜力，但也面临以形式化方式建模和管理意见的挑战。本文介绍了一种通过意见来引导强化学习代理的方法，并提供了一种端到端的方法来建模和管理顾问的意见。通过合成和人类顾问以及不同级别的不确定性和多个建议策略的评估，我们评估了该方法的效用。结果表明，即使是不确定的意见也能提高强化学习代理的性能，从而获得更高的奖励、更高效的探索和更好的加强策略。虽然我们在一个简化的拓扑运行示例中演示了我们的方法，但我们的方法也适用于具有更高维度的复杂问题。

意见引导的强化学习

Opinion-Guided Reinforcement Learning

A longstanding goal of artificial intelligence is to create artificial agents
capable of learning to perform tasks that require sequential decision making.
Importantly, while it is the artificial agent that learns and acts, it is still
up to humans to specify the particular task to be performed. Classical
task-specification approaches typically involve humans providing stationary
reward functions or explicit demonstrations of the desired tasks. However,
there has recently been a great deal of research energy invested in exploring
alternative ways in which humans may guide learning agents that may, e.g., be
more suitable for certain tasks or require less human effort. This survey
provides a high-level overview of five recent machine learning frameworks that
primarily rely on human guidance apart from pre-specified reward functions or
conventional, step-by-step action demonstrations. We review the motivation,
assumptions, and implementation of each framework, and we discuss possible
future research directions.

本文综述了五种最近主要依赖于人类指导的机器学习框架及其动机，假设和实施，并讨论可能的未来研究方向。

利用人类辅助完成顺序决策任务的最新进展

Recent Advances in Leveraging Human Guidance for Sequential  Decision-Making Tasks

Reinforcement learning agents can learn to solve sequential decision tasks by
interacting with the environment. Human knowledge of how to solve these tasks
can be incorporated using imitation learning, where the agent learns to imitate
human demonstrated decisions. However, human guidance is not limited to the
demonstrations. Other types of guidance could be more suitable for certain
tasks and require less human effort. This survey provides a high-level overview
of five recent learning frameworks that primarily rely on human guidance other
than conventional, step-by-step action demonstrations. We review the
motivation, assumption, and implementation of each framework. We then discuss
possible future research directions.

本文综述了五个最近的学习框架，这些框架主要依赖人类指导，而不是传统的一步一步的动作演示，评估了每个框架的动机、假设和实现，并讨论了可能的未来研究方向。