Transformer-based models trained on large and general purpose datasets
consisting of molecular strings have recently emerged as a powerful tool for
successfully modeling various structure-property relations. Inspired by this
success, we extend the paradigm of training chemical language transformers on
large-scale chemical datasets to generative tasks in this work. Specifically,
we propose GP-MoLFormer, an autoregressive molecular string generator that is
trained on more than 1.1B chemical SMILES. GP-MoLFormer uses a 46.8M parameter
transformer decoder model with linear attention and rotary positional encodings
as the base architecture. We explore the utility of GP-MoLFormer in generating
novel, valid, and unique SMILES. Impressively, we find GP-MoLFormer is able to
generate a significant fraction of novel, valid, and unique SMILES even when
the number of generated molecules is in the 10 billion range and the reference
set is over a billion. We also find strong memorization of training data in
GP-MoLFormer generations, which has so far remained unexplored for chemical
language models. Our analyses reveal that training data memorization and
novelty in generations are impacted by the quality of the training data;
duplication bias in training data can enhance memorization at the cost of
lowering novelty. We evaluate GP-MoLFormer's utility and compare it with that
of existing baselines on three different tasks: de novo generation,
scaffold-constrained molecular decoration, and unconstrained property-guided
optimization. While the first two are handled with no additional training, we
propose a parameter-efficient fine-tuning method for the last task, which uses
property-ordered molecular pairs as input. We call this new approach
pair-tuning. Our results show GP-MoLFormer performs better or comparable with
baselines across all three tasks, demonstrating its general utility.

训练数据记忆和新颖性受训练数据质量影响；GP-MoLFormer 可用于生成新颖、有效且独特的分子；在三个不同任务上，GP-MoLFormer 表现良好或相当，展示了它的通用性。