Protein language models are a powerful tool for learning protein
representations through pre-training on vast protein sequence datasets.
However, traditional protein language models lack explicit structural
supervision, despite its relevance to protein function. To address this issue,
we introduce the integration of remote homology detection to distill structural
information into protein language models without requiring explicit protein
structures as input. We evaluate the impact of this structure-informed training
on downstream protein function prediction tasks. Experimental results reveal
consistent improvements in function annotation accuracy for EC number and GO
term prediction. Performance on mutant datasets, however, varies based on the
relationship between targeted properties and protein structures. This
underscores the importance of considering this relationship when applying
structure-aware training to protein function prediction tasks. Code and model
weights are available at this https URL

通过将远程同源检测集成到蛋白质语言模型中，我们实现了将结构信息融入蛋白质语言模型，从而在蛋白质功能预测任务中提高了功能注释准确性。

基于结构信息的蛋白质语言模型

Structure-Informed Protein Language Model

Prior work has shown that structural supervision helps English language
models learn generalizations about syntactic phenomena such as subject-verb
agreement. However, it remains unclear if such an inductive bias would also
improve language models' ability to learn grammatical dependencies in
typologically different languages. Here we investigate this question in
Mandarin Chinese, which has a logographic, largely syllable-based writing
system; different word order; and sparser morphology than English. We train
LSTMs, Recurrent Neural Network Grammars, Transformer language models, and
Transformer-parameterized generative parsing models on two Mandarin Chinese
datasets of different sizes. We evaluate the models' ability to learn different
aspects of Mandarin grammar that assess syntactic and semantic relationships.
We find suggestive evidence that structural supervision helps with representing
syntactic state across intervening content and improves performance in low-data
settings, suggesting that the benefits of hierarchical inductive biases in
acquiring dependency relationships may extend beyond English.

我们研究了结构化监督在汉语中的效果，结果表明结构化监督可以帮助语言模型学习句法和语义关系，并且在低数据情况下表现出更好的性能。

汉语语言模型中语法知识的受控评估

Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese  Language Models

We present a novel Dynamic Differentiable Reasoning (DDR) framework for
jointly learning branching programs and the functions composing them; this
resolves a significant nondifferentiability inhibiting recent dynamic
architectures. We apply our framework to two settings in two highly compact and
data efficient architectures: DDRprog for CLEVR Visual Question Answering and
DDRstack for reverse Polish notation expression evaluation. DDRprog uses a
recurrent controller to jointly predict and execute modular neural programs
that directly correspond to the underlying question logic; it explicitly forks
subprocesses to handle logical branching. By effectively leveraging additional
structural supervision, we achieve a large improvement over previous approaches
in subtask consistency and a small improvement in overall accuracy. We further
demonstrate the benefits of structural supervision in the RPN setting: the
inclusion of a stack assumption in DDRstack allows our approach to generalize
to long expressions where an LSTM fails the task.

介绍了一种新颖的动态可微推理（DDR）框架，它可用于学习分支程序和其组成部分的函数，解决了最近动态架构中的重要非可微问题，并将其应用于 CLEVR 可视化问题回答和逆波兰表达式评估的两个设置中，通过有效利用额外的结构监督，在子任务一致性方面实现了大的改进和总体准确性的小提高。