The Generative Flow Network (GFlowNet) is a probabilistic framework in which
an agent learns a stochastic policy and flow functions to sample objects with
probability proportional to an unnormalized reward function. GFlowNets share a
strong resemblance to reinforcement learning (RL), that typically aims to
maximize reward, due to their sequential decision-making processes. Recent
works have studied connections between GFlowNets and maximum entropy (MaxEnt)
RL, which modifies the standard objective of RL agents by learning an
entropy-regularized objective. However, a critical theoretical gap persists:
despite the apparent similarities in their sequential decision-making nature, a
direct link between GFlowNets and standard RL has yet to be discovered, while
bridging this gap could further unlock the potential of both fields. In this
paper, we establish a new connection between GFlowNets and policy evaluation
for a uniform policy. Surprisingly, we find that the resulting value function
for the uniform policy has a close relationship to the flows in GFlowNets.
Leveraging these insights, we further propose a novel rectified policy
evaluation (RPE) algorithm, which achieves the same reward-matching effect as
GFlowNets, offering a new perspective. We compare RPE, MaxEnt RL, and GFlowNets
in a number of benchmarks, and show that RPE achieves competitive results
compared to previous approaches. This work sheds light on the previously
unexplored connection between (non-MaxEnt) RL and GFlowNets, potentially
opening new avenues for future research in both fields.

生成流网络（GFlowNet）是一个概率框架，其中代理通过学习随机策略和流函数，以与未归一化奖励函数成比例的概率进行对象采样。在这篇论文中，我们建立了 GFlowNet 与均匀策略的策略评估之间的新联系，并提出了一种新颖的修正型策略评估（RPE）算法，同时比较了 RPE、MaxEnt RL 和 GFlowNets 在多个基准测试下的性能。这项工作为（非 MaxEnt）RL 与 GFlowNets 之间的未曾探索的联系提供了新的启示，可能为两个领域的未来研究开辟了新的途径。

弥补强化学习中奖励匹配问题

Rectifying Reinforcement Learning for Reward Matching

We seek to automate the generation of drug-like compounds conditioned to
specific protein pocket targets. Most current methods approximate the
protein-molecule distribution of a finite dataset and, therefore struggle to
generate molecules with significant binding improvement over the training
dataset. We instead frame the pocket-conditioned molecular generation task as
an RL problem and develop TacoGFN, a target conditional Generative Flow Network
model. Our method is explicitly encouraged to generate molecules with desired
properties as opposed to fitting on a pre-existing data distribution. To this
end, we develop transformer-based docking score prediction to speed up docking
score computation and propose TacoGFN to explore molecule space efficiently.
Furthermore, we incorporate several rounds of active learning where generated
samples are queried using a docking oracle to improve the docking score
prediction. This approach allows us to accurately explore as much of the
molecule landscape as we can afford computationally. Empirically, molecules
generated using TacoGFN and its variants significantly outperform all baseline
methods across every property (Docking score, QED, SA, Lipinski), while being
orders of magnitude faster.

我们致力于自动化生成与特定蛋白质口袋靶点相匹配的药物样化合物。我们采用强化学习方法，提出了一种目标条件生成流网络（TacoGFN）模型，通过该模型可以生成具有所需属性的分子。我们还开发了基于 Transformer 的对接评分预测方法，用于加速对接评分计算，并使用几轮主动学习来改善对接评分预测。实验证明，使用 TacoGFN 及其变种生成的分子在每个特性（对接评分，QED，SA，Lipinski）上显著优于所有基准方法，并且速度快了数个数量级。