Three-dimensional native states of natural proteins display recurring and
hierarchical patterns. Yet, traditional graph-based modeling of protein
structures is often limited to operate within a single fine-grained resolution,
and lacks hourglass neural architectures to learn those high-level building
blocks. We narrow this gap by introducing Ophiuchus, an SO(3)-equivariant
coarse-graining model that efficiently operates on all heavy atoms of standard
protein residues, while respecting their relevant symmetries. Our model departs
from current approaches that employ graph modeling, instead focusing on local
convolutional coarsening to model sequence-motif interactions in log-linear
length complexity. We train Ophiuchus on contiguous fragments of PDB monomers,
investigating its reconstruction capabilities across different compression
rates. We examine the learned latent space and demonstrate its prompt usage in
conformational interpolation, comparing interpolated trajectories to structure
snapshots from the PDBFlex dataset. Finally, we leverage denoising diffusion
probabilistic models (DDPM) to efficiently sample readily-decodable latent
embeddings of diverse miniproteins. Our experiments demonstrate Ophiuchus to be
a scalable basis for efficient protein modeling and generation.

通过引入 Ophiuchus，一种 SO（3）- 等变粗粒化模型，我们在所有标准蛋白质残基的重原子上高效地操作，同时尊重其相关的对称性，从而缩小这个差距，并采用局部卷积粗化模型来模拟序列模体间的相互作用。我们训练 Ophiuchus 来重建 PDB 单体的连续片段，研究其在不同压缩率下的重建能力，并检查学习到的潜在空间，演示其在构象内插中的快速应用，并将插值轨迹与 PDBFlex 数据集中的结构快照进行比较。最后，我们利用去噪扩散概率模型（DDPM）高效地采样多样的小型蛋白质可直接解码的潜在嵌入。我们的实验表明 Ophiuchus 是一种可扩展的有效蛋白质建模和生成基础。

摘星座：通过分层粗化的 SO (3) 等变自编码器实现蛋白质结构的可扩展建模

Ophiuchus: Scalable Modeling of Protein Structures through Hierarchical  Coarse-graining SO(3)-Equivariant Autoencoders

Attention, specifically scaled dot-product attention, has proven effective
for natural language, but it does not have a mechanism for handling
hierarchical patterns of arbitrary nesting depth, which limits its ability to
recognize certain syntactic structures. To address this shortcoming, we propose
stack attention: an attention operator that incorporates stacks, inspired by
their theoretical connections to context-free languages (CFLs). We show that
stack attention is analogous to standard attention, but with a latent model of
syntax that requires no syntactic supervision. We propose two variants: one
related to deterministic pushdown automata (PDAs) and one based on
nondeterministic PDAs, which allows transformers to recognize arbitrary CFLs.
We show that transformers with stack attention are very effective at learning
CFLs that standard transformers struggle on, achieving strong results on a CFL
with theoretically maximal parsing difficulty. We also show that stack
attention is more effective at natural language modeling under a constrained
parameter budget, and we include results on machine translation.

我们提出了堆栈注意力：一种集成堆栈的注意力操作符，其灵感来源于堆栈与上下文无关语言（CFLs）的理论联系，以解决标准注意力无法识别某些句法结构的局限性。我们提出了两种变种：一种与确定性下推自动机（PDAs）相关，一种基于非确定性 PDAs，使得 transformers 能够识别任意 CFLs。我们证明，具有堆栈注意力的 transformers 在学习 CFLs 方面非常有效，能够在理论上具有最大的解析困难的 CFL 上取得强大的结果。我们还证明，堆栈注意力在受约束参数预算下在自然语言建模中更加有效，并包括机器翻译方面的结果。

层叠注意力：提高 Transformer 模型对层次模式建模的能力

Stack Attention: Improving the Ability of Transformers to Model  Hierarchical Patterns

Graph data are pervasive in many real-world applications. Recently,
increasing attention has been paid on graph neural networks (GNNs), which aim
to model the local graph structures and capture the hierarchical patterns by
aggregating the information from neighbors with stackable network modules.
Motivated by the observation that different nodes often require different
iterations of aggregation to fully capture the structural information, in this
paper, we propose to explicitly sample diverse iterations of aggregation for
different nodes to boost the performance of GNNs. It is a challenging task to
develop an effective aggregation strategy for each node, given complex graphs
and sparse features. Moreover, it is not straightforward to derive an efficient
algorithm since we need to feed the sampled nodes into different number of
network layers. To address the above challenges, we propose Policy-GNN, a
meta-policy framework that models the sampling procedure and message passing of
GNNs into a combined learning process. Specifically, Policy-GNN uses a
meta-policy to adaptively determine the number of aggregations for each node.
The meta-policy is trained with deep reinforcement learning (RL) by exploiting
the feedback from the model. We further introduce parameter sharing and a
buffer mechanism to boost the training efficiency. Experimental results on
three real-world benchmark datasets suggest that Policy-GNN significantly
outperforms the state-of-the-art alternatives, showing the promise in
aggregation optimization for GNNs.

本文提出了 Policy-GNN 方法，通过 meta-policy 学习采样策略和消息传递，以解决 GNN 中的聚合优化问题，提高对图数据的建模精度。实验表明 Policy-GNN 显著优于现有的替代方法。