While off-policy reinforcement learning (RL) algorithms are sample efficient
due to gradient-based updates and data reuse in the replay buffer, they
struggle with convergence to local optima due to limited exploration. On the
other hand, population-based algorithms offer a natural exploration strategy,
but their heuristic black-box operators are inefficient. Recent algorithms have
integrated these two methods, connecting them through a shared replay buffer.
However, the effect of using diverse data from population optimization
iterations on off-policy RL algorithms has not been thoroughly investigated. In
this paper, we first analyze the use of off-policy RL algorithms in combination
with population-based algorithms, showing that the use of population data could
introduce an overlooked error and harm performance. To test this, we propose a
uniform and scalable training design and conduct experiments on our tailored
framework in robot locomotion tasks from the OpenAI gym. Our results
substantiate that using population data in off-policy RL can cause instability
during training and even degrade performance. To remedy this issue, we further
propose a double replay buffer design that provides more on-policy data and
show its effectiveness through experiments. Our results offer practical
insights for training these hybrid methods.

本文深入研究了结合 off-policy reinforcement learning 和 population-based optimization 两种算法的训练方法，在机器人运动任务中的实验结果表明 population data 的使用会影响训练稳定性并降低性能，作者进一步提出了双重回放缓冲设计来解决此问题。

重新思考基于群体协助的离线策略强化学习

Rethinking Population-assisted Off-policy Reinforcement Learning

We study an important and challenging task of attacking natural language
processing models in a hard label black box setting. We propose a
decision-based attack strategy that crafts high quality adversarial examples on
text classification and entailment tasks. Our proposed attack strategy
leverages population-based optimization algorithm to craft plausible and
semantically similar adversarial examples by observing only the top label
predicted by the target model. At each iteration, the optimization procedure
allow word replacements that maximizes the overall semantic similarity between
the original and the adversarial text. Further, our approach does not rely on
using substitute models or any kind of training data. We demonstrate the
efficacy of our proposed approach through extensive experimentation and
ablation studies on five state-of-the-art target models across seven benchmark
datasets. In comparison to attacks proposed in prior literature, we are able to
achieve a higher success rate with lower word perturbation percentage that too
in a highly restricted setting.

该研究提出了一种决策式的攻击策略，利用基于人口统计的优化算法，通过仅观察目标模型预测的前一标签，制作出可信且语义相似的对抗性例子。与先前文献中提出的攻击相比，在高度限制的情况下，成功率更高，被替换单词比例更低。