A growing trend involves integrating human knowledge into learning
frameworks, leveraging subtle human feedback to refine AI models. Despite these
advances, no comprehensive theoretical framework describing the specific
conditions under which human comparisons improve the traditional supervised
fine-tuning process has been developed. To bridge this gap, this paper studies
the effective use of human comparisons to address limitations arising from
noisy data and high-dimensional models. We propose a two-stage "Supervised Fine
Tuning+Human Comparison" (SFT+HC) framework connecting machine learning with
human feedback through a probabilistic bisection approach. The two-stage
framework first learns low-dimensional representations from noisy-labeled data
via an SFT procedure, and then uses human comparisons to improve the model
alignment. To examine the efficacy of the alignment phase, we introduce a novel
concept termed the "label-noise-to-comparison-accuracy" (LNCA) ratio. This
paper theoretically identifies the conditions under which the "SFT+HC"
framework outperforms pure SFT approach, leveraging this ratio to highlight the
advantage of incorporating human evaluators in reducing sample complexity. We
validate that the proposed conditions for the LNCA ratio are met in a case
study conducted via an Amazon Mechanical Turk experiment.

探究人类比较对传统监督微调过程的改善条件，提出一种连接机器学习和人类反馈的两阶段框架，并利用概率二分法通过学习低维表示和使用人类比较来改善模型对齐，进而减少样本复杂度，实验证明该框架在降噪数据和高维模型中的优势。

一种基于人工比较的概率对齐方法

A Probabilistic Approach for Alignment with Human Comparisons

Reinforcement learning agents can learn to solve sequential decision tasks by
interacting with the environment. Human knowledge of how to solve these tasks
can be incorporated using imitation learning, where the agent learns to imitate
human demonstrated decisions. However, human guidance is not limited to the
demonstrations. Other types of guidance could be more suitable for certain
tasks and require less human effort. This survey provides a high-level overview
of five recent learning frameworks that primarily rely on human guidance other
than conventional, step-by-step action demonstrations. We review the
motivation, assumption, and implementation of each framework. We then discuss
possible future research directions.

本文综述了五个最近的学习框架，这些框架主要依赖人类指导，而不是传统的一步一步的动作演示，评估了每个框架的动机、假设和实现，并讨论了可能的未来研究方向。

利用人类指导提升深度强化学习任务

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

For a seamless deployment of the Internet of Things (IoT), there is a need
for self-organizing solutions to overcome key IoT challenges that include data
processing, resource management, coexistence with existing wireless networks,
and improved IoT-wide event detection. One of the most promising solutions to
address these challenges is via the use of innovative learning frameworks that
will enable the IoT devices to operate autonomously in a dynamic environment.
However, developing learning mechanisms for the IoT requires coping with unique
IoT properties in terms of resource constraints, heterogeneity, and strict
quality-of-service requirements. In this paper, a number of emerging learning
frameworks suitable for IoT applications are presented. In particular, the
advantages, limitations, IoT applications, and key results pertaining to
machine learning, sequential learning, and reinforcement learning are studied.
For each type of learning, the computational complexity, required information,
and learning performance are discussed. Then, to handle the heterogeneity of
the IoT, a new framework based on the powerful tools of cognitive hierarchy
theory is introduced. This framework is shown to efficiently capture the
different IoT device types and varying levels of available resources among the
IoT devices. In particular, the different resource capabilities of IoT devices
are mapped to different levels of rationality in cognitive hierarchy theory,
thus enabling the IoT devices to use different learning frameworks depending on
their available resources. Finally, key results on the use of cognitive
hierarchy theory in the IoT are presented.

本文介绍了适用于物联网应用的多种学习框架，分别研究了机器学习、顺序学习和强化学习的优缺点、适用性及关键结果，并引入基于认知层次理论的新框架来处理物联网的异构性和资源限制，最终展示了在物联网中使用认知层次理论的关键结果。