Protein sequence design, determined by amino acid sequences, are essential to
protein engineering problems in drug discovery. Prior approaches have resorted
to evolutionary strategies or Monte-Carlo methods for protein design, but often
fail to exploit the structure of the combinatorial search space, to generalize
to unseen sequences. In the context of discrete black box optimization over
large search spaces, learning a mutation policy to generate novel sequences
with reinforcement learning is appealing. Recent advances in protein language
models (PLMs) trained on large corpora of protein sequences offer a potential
solution to this problem by scoring proteins according to their biological
plausibility (such as the TM-score). In this work, we propose to use PLMs as a
reward function to generate new sequences. Yet the PLM can be computationally
expensive to query due to its large size. To this end, we propose an
alternative paradigm where optimization can be performed on scores from a
smaller proxy model that is periodically finetuned, jointly while learning the
mutation policy. We perform extensive experiments on various sequence lengths
to benchmark RL-based approaches, and provide comprehensive evaluations along
biological plausibility and diversity of the protein. Our experimental results
include favorable evaluations of the proposed sequences, along with high
diversity scores, demonstrating that RL is a strong candidate for biological
sequence design. Finally, we provide a modular open source implementation can
be easily integrated in most RL training loops, with support for replacing the
reward model with other PLMs, to spur further research in this domain. The code
for all experiments is provided in the supplementary material.

蛋白质序列设计对于药物研发中的蛋白质工程问题至关重要，本研究提出使用蛋白质语言模型作为奖励函数来生成新的序列，利用强化学习和最优化方法进行体系建模，实验证明强化学习在生物序列设计方面具有很大潜力。

利用蛋白质语言模型进行序列设计的强化学习

Reinforcement Learning for Sequence Design Leveraging Protein Language  Models

The design of novel protein sequences with targeted functionalities underpins
a central theme in protein engineering, impacting diverse fields such as drug
discovery and enzymatic engineering. However, navigating this vast
combinatorial search space remains a severe challenge due to time and financial
constraints. This scenario is rapidly evolving as the transformative
advancements in AI, particularly in the realm of generative models and
optimization algorithms, have been propelling the protein design field towards
an unprecedented revolution. In this survey, we systematically review recent
advances in generative AI for controllable protein sequence design. To set the
stage, we first outline the foundational tasks in protein sequence design in
terms of the constraints involved and present key generative models and
optimization algorithms. We then offer in-depth reviews of each design task and
discuss the pertinent applications. Finally, we identify the unresolved
challenges and highlight research opportunities that merit deeper exploration.

通过综述最新的生成式人工智能在可控蛋白序列设计方面的进展，本文提出了蛋白质工程、生成式人工智能、蛋白序列设计、优化算法和研究机会等关键词，以及未解决的挑战。

可控蛋白质序列设计的生成式人工智能：综述

Generative AI for Controllable Protein Sequence Design: A Survey

Protein language models (pLMs), pre-trained via causal language modeling on
protein sequences, have been a promising tool for protein sequence design. In
real-world protein engineering, there are many cases where the amino acids in
the middle of a protein sequence are optimized while maintaining other
residues. Unfortunately, because of the left-to-right nature of pLMs, existing
pLMs modify suffix residues by prompting prefix residues, which are
insufficient for the infilling task that considers the whole surrounding
context. To find the more effective pLMs for protein engineering, we design a
new benchmark, Secondary structureE InFilling rEcoveRy, SEIFER, which
approximates infilling sequence design scenarios. With the evaluation of
existing models on the benchmark, we reveal the weakness of existing language
models and show that language models trained via fill-in-middle transformation,
called ProtFIM, are more appropriate for protein engineering. Also, we prove
that ProtFIM generates protein sequences with decent protein representations
through exhaustive experiments and visualizations.

通过设计一个新的基准（SEIFER）来评估现有的蛋白质语言模型，研究显示，通过中间填充转换（ProtFIM）训练的语言模型更适合蛋白质工程，并能够产生良好的蛋白质表示。