Reinforcement learning from human feedback (RLHF) has contributed to
performance improvements in large language models. To tackle its reliance on
substantial amounts of human-labeled data, a successful approach is multi-task
representation learning, which involves learning a high-quality,
low-dimensional representation from a wide range of source tasks. In this
paper, we formulate RLHF as the contextual dueling bandit problem and assume a
common linear representation. We demonstrate that the sample complexity of
source tasks in multi-task RLHF can be reduced by considering task relevance
and allocating different sample sizes to source tasks with varying task
relevance. We further propose an algorithm to estimate task relevance by a
small number of additional data and then learn a policy. We prove that to
achieve $\varepsilon-$optimal, the sample complexity of the source tasks can be
significantly reduced compared to uniform sampling. Additionally, the sample
complexity of the target task is only linear in the dimension of the latent
space, thanks to representation learning.

通过多任务表示学习的方式，我们将人类反馈强化学习 (RLHF) 问题建模为一种上下文二分问题，并假设存在一种共同线性表示。我们证明了考虑任务相关性，并为具有不同任务相关性的源任务分配不同样本数量可以降低多任务 RLHF 中的样本复杂度。此外，由于表示学习，目标任务的样本复杂度仅与潜在空间的维度成线性关系。

强化学习中主动多任务学习的力量

The Power of Active Multi-Task Learning in Reinforcement Learning from  Human Feedback

Pruning large neural networks while maintaining their performance is often
desirable due to the reduced space and time complexity. In existing methods,
pruning is done within an iterative optimization procedure with either
heuristically designed pruning schedules or additional hyperparameters,
undermining their utility. In this work, we present a new approach that prunes
a given network once at initialization prior to training. To achieve this, we
introduce a saliency criterion based on connection sensitivity that identifies
structurally important connections in the network for the given task. This
eliminates the need for both pretraining and the complex pruning schedule while
making it robust to architecture variations. After pruning, the sparse network
is trained in the standard way. Our method obtains extremely sparse networks
with virtually the same accuracy as the reference network on the MNIST,
CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to
various architectures including convolutional, residual and recurrent networks.
Unlike existing methods, our approach enables us to demonstrate that the
retained connections are indeed relevant to the given task.

本文介绍了一种通过基于连接灵敏度的显著性标准在初始化前修剪神经网络，从而在保持网络性能的同时减少空间和时间复杂度的新方法，并应用于 MNIST、CIFAR-10 和 Tiny-ImageNet 分类任务的各种体系结构中，所得到的极度稀疏网络的准确性与基准网络几乎相同，而且能够证明保留的连接确实与给定任务相关。

SNIP：基于连接敏感度的单次剪枝神经网络

SNIP: Single-shot Network Pruning based on Connection Sensitivity

Existing models which generate textual explanations enforce task relevance
through a discriminative term loss function, but such mechanisms only weakly
constrain mentioned object parts to actually be present in the image. In this
paper, a new model is proposed for generating explanations by utilizing
localized grounding of constituent phrases in generated explanations to ensure
image relevance. Specifically, we introduce a phrase-critic model to refine
(re-score/re-rank) generated candidate explanations and employ a
relative-attribute inspired ranking loss using "flipped" phrases as negative
examples for training. At test time, our phrase-critic model takes an image and
a candidate explanation as input and outputs a score indicating how well the
candidate explanation is grounded in the image.

本论文提出了一种新的模型，通过利用在生成的解释中的成分词的本地化接地来确保图像相关性，从而同时生成文本解释和图像。