Transformer architectures have proven to learn useful representations for
protein classification and generation tasks. However, these representations
present challenges in interpretability. In this work, we demonstrate a set of
methods for analyzing protein Transformer models through the lens of attention.
We show that attention: (1) captures the folding structure of proteins,
connecting amino acids that are far apart in the underlying sequence, but
spatially close in the three-dimensional structure, (2) targets binding sites,
a key functional component of proteins, and (3) focuses on progressively more
complex biophysical properties with increasing layer depth. We find this
behavior to be consistent across three Transformer architectures (BERT, ALBERT,
XLNet) and two distinct protein datasets. We also present a three-dimensional
visualization of the interaction between attention and protein structure. Code
for visualization and analysis is available at
this https URL

本研究通过注意力机制探索分析蛋白质 Transformer 模型，展示其捕捉蛋白质折叠结构，定位蛋白质功能性区域，以及随着层数增加逐渐关注复杂生物物理特性等行为，结果表明三种 Transformer 架构对此表现一致，并提供蛋白质结构和注意力之间的三维可视化交互。

BERT 与生物学相遇：解析蛋白质语言模型中的注意力

BERTology Meets Biology: Interpreting Attention in Protein Language  Models

Substring kernels are classical tools for representing biological sequences
or text. However, when large amounts of annotated data are available, models
that allow end-to-end training such as neural networks are often preferred.
Links between recurrent neural networks (RNNs) and substring kernels have
recently been drawn, by formally showing that RNNs with specific activation
functions were points in a reproducing kernel Hilbert space (RKHS). In this
paper, we revisit this link by generalizing convolutional kernel
networks---originally related to a relaxation of the mismatch kernel---to model
gaps in sequences. It results in a new type of recurrent neural network which
can be trained end-to-end with backpropagation, or without supervision by using
kernel approximation techniques. We experimentally show that our approach is
well suited to biological sequences, where it outperforms existing methods for
protein classification tasks.

本文探讨了 RNN 与 Substring kernel 之间的联系，并将卷积核网络与序列间的间隙相关联，推出了一种新的用于蛋白质分类的递归神经网络，允许端到端的训练，实验结果表明与现有方法相比，我们的方法更加适用于生物序列。

循环核网络

Recurrent Kernel Networks

Protein function and dynamics are closely related to its sequence and
structure. However prediction of protein function and dynamics from its
sequence and structure is still a fundamental challenge in molecular biology.
Protein classification, which is typically done through measuring the
similarity be- tween proteins based on protein sequence or physical
information, serves as a crucial step toward the understanding of protein
function and dynamics. Persistent homology is a new branch of algebraic
topology that has found its success in the topological data analysis in a
variety of disciplines, including molecular biology. The present work explores
the potential of using persistent homology as an indepen- dent tool for protein
classification. To this end, we propose a molecular topological fingerprint
based support vector machine (MTF-SVM) classifier. Specifically, we construct
machine learning feature vectors solely from protein topological fingerprints,
which are topological invariants generated during the filtration process. To
validate the present MTF-SVM approach, we consider four types of problems.
First, we study protein-drug binding by using the M2 channel protein of
influenza A virus. We achieve 96% accuracy in discriminating drug bound and
unbound M2 channels. Additionally, we examine the use of MTF-SVM for the
classification of hemoglobin molecules in their relaxed and taut forms and
obtain about 80% accuracy. The identification of all alpha, all beta, and
alpha-beta protein domains is carried out in our next study using 900 proteins.
We have found a 85% success in this identifica- tion. Finally, we apply the
present technique to 55 classification tasks of protein superfamilies over 1357
samples. An average accuracy of 82% is attained. The present study establishes
computational topology as an independent and effective alternative for protein
classification.

本研究探讨了使用持久同调作为蛋白质分类的独立工具的潜力，通过构建机器学习特征向量仅基于蛋白质拓扑指纹，提出了一种基于分子拓扑指纹的支持向量机分类器，并取得了较高的分类精度。

蛋白质分类的拓扑学方法

A topological approach for protein classification

We introduce a new nearest-prototype classifier, the prototype vector machine
(PVM). It arises from a combinatorial optimization problem which we cast as a
variant of the set cover problem. We propose two algorithms for approximating
its solution. The PVM selects a relatively small number of representative
points which can then be used for classification. It contains 1-NN as a special
case. The method is compatible with any dissimilarity measure, making it
amenable to situations in which the data are not embedded in an underlying
feature space or in which using a non-Euclidean metric is desirable. Indeed, we
demonstrate on the much studied ZIP code data how the PVM can reap the benefits
of a problem-specific metric. In this example, the PVM outperforms the highly
successful 1-NN with tangent distance, and does so retaining fewer than half of
the data points. This example highlights the strengths of the PVM in yielding a
low-error, highly interpretable model. Additionally, we apply the PVM to a
protein classification problem in which a kernel-based distance is used.

介绍了一种新的最近原型分类器，即原型向量机 (PVM)，它选择相对较少的代表点用于分类，可适用任何差异测量，并适用于非欧几里德度量的情况，更好地解决了一些特定问题，并适用于蛋白质分类问题。