GNNs and chemical fingerprints are the predominant approaches to representing
molecules for property prediction. However, in NLP, transformers have become
the de-facto standard for representation learning thanks to their strong
downstream task transfer. In parallel, the software ecosystem around
transformers is maturing rapidly, with libraries like HuggingFace and BertViz
enabling streamlined training and introspection. In this work, we make one of
the first attempts to systematically evaluate transformers on molecular
property prediction tasks via our ChemBERTa model. ChemBERTa scales well with
pretraining dataset size, offering competitive downstream performance on
MoleculeNet and useful attention-based visualization modalities. Our results
suggest that transformers offer a promising avenue of future work for molecular
representation learning and property prediction. To facilitate these efforts,
we release a curated dataset of 77M SMILES from PubChem suitable for
large-scale self-supervised pretraining.

这项工作通过 ChemBERTa 模型对分子性质预测任务中的 transformers 进行了系统评估，结果表明 transformers 在分子表示学习和性质预测方面具有很大的发展潜力，同时我们为大规模自监督预训练提供了一份 77M SMILES 的数据集。