Transformers are the type of neural networks that has revolutionised natural
language processing and protein science. Their key building block is a
mechanism called self-attention which is trained to predict missing words in
sentences. Despite the practical success of transformers in applications it
remains unclear what self-attention learns from data, and how. Here, we give a
precise analytical and numerical characterisation of transformers trained on
data drawn from a generalised Potts model with interactions between sites and
Potts colours. While an off-the-shelf transformer requires several layers to
learn this distribution, we show analytically that a single layer of
self-attention with a small modification can learn the Potts model exactly in
the limit of infinite sampling. We show that this modified self-attention, that
we call ``factored'', has the same functional form as the conditional
probability of a Potts spin given the other spins, compute its generalisation
error using the replica method from statistical physics, and derive an exact
mapping to pseudo-likelihood methods for solving the inverse Ising and Potts
problem.

通过对来自一个广义 Potts 模型的数据进行学习，我们证明了带一点修改的自注意力单层可以在无限采样的极限下精确地学习这个分布，这种修改后的自注意力具有与条件概率相同的功能形式。

利用分解注意力的单层变换器对广义 Potts 模型进行最优推断

Optimal inference of a generalised Potts model by single-layer  transformers with factored attention

Existing data-centric methods for protein science generally cannot
sufficiently capture and leverage biology knowledge, which may be crucial for
many protein tasks. To facilitate research in this field, we create
ProteinKG65, a knowledge graph for protein science. Using gene ontology and
Uniprot knowledge base as a basis, we transform and integrate various kinds of
knowledge with aligned descriptions and protein sequences, respectively, to GO
terms and protein entities. ProteinKG65 is mainly dedicated to providing a
specialized protein knowledge graph, bringing the knowledge of Gene Ontology to
protein function and structure prediction. We also illustrate the potential
applications of ProteinKG65 with a prototype. Our dataset can be downloaded at
this https URL.

本文介绍了 ProteinKG65，一个专门用于蛋白质科学的知识图谱，它基于基因本体论和 Uniprot 知识库，并整合了多种知识，主要旨在将基因本体论的知识应用于蛋白质功能及结构预测，文中还展示了 ProteinKG65 的潜在应用。