Clinical variant classification of pathogenic versus benign genetic variants
remains a challenge in clinical genetics. Recently, the proposition of genomic
foundation models has improved the generic variant effect prediction (VEP)
accuracy via weakly-supervised or unsupervised training. However, these VEPs
are not disease-specific, limiting their adaptation at the point of care. To
address this problem, we propose DYNA: Disease-specificity fine-tuning via a
Siamese neural network broadly applicable to all genomic foundation models for
more effective variant effect predictions in disease-specific contexts. We
evaluate DYNA in two distinct disease-relevant tasks. For coding VEPs, we focus
on various cardiovascular diseases, where gene-disease relationships of
loss-of-function vs. gain-of-function dictate disease-specific VEP. For
non-coding VEPs, we apply DYNA to an essential post-transcriptional regulatory
axis of RNA splicing, the most common non-coding pathogenic mechanism in
established clinical VEP guidelines. In both cases, DYNA fine-tunes various
pre-trained genomic foundation models on small, rare variant sets. The DYNA
fine-tuned models show superior performance in the held-out rare variant
testing set and are further replicated in large, clinically-relevant variant
annotations in ClinVAR. Thus, DYNA offers a potent disease-specific variant
effect prediction method, excelling in intra-gene generalization and
generalization to unseen genetic variants, making it particularly valuable for
disease associations and clinical applicability.

通过提出 DYNA 方法，通过 Siamese 神经网络针对所有基因组学基础模型进行疾病特异性微调，从而提高了变异效应预测的准确性，这对疾病关联和临床适用性具有重要价值。

DYNA：用于变异致病性的疾病特定语言模型

DYNA: Disease-Specific Language Model for Variant Pathogenicity

Model-based reinforcement learning has been empirically demonstrated as a
successful strategy to improve sample efficiency. In particular, Dyna is an
elegant model-based architecture integrating learning and planning that
provides huge flexibility of using a model. One of the most important
components in Dyna is called search-control, which refers to the process of
generating state or state-action pairs from which we query the model to acquire
simulated experiences. Search-control is critical in improving learning
efficiency. In this work, we propose a simple and novel search-control strategy
by searching high frequency regions of the value function. Our main intuition
is built on Shannon sampling theorem from signal processing, which indicates
that a high frequency signal requires more samples to reconstruct. We
empirically show that a high frequency function is more difficult to
approximate. This suggests a search-control strategy: we should use states from
high frequency regions of the value function to query the model to acquire more
samples. We develop a simple strategy to locally measure the frequency of a
function by gradient and hessian norms, and provide theoretical justification
for this approach. We then apply our strategy to search-control in Dyna, and
conduct experiments to show its property and effectiveness on benchmark
domains.

本文提出了一种基于梯度和海森矩阵范数的频率度量策略，通过对价值函数高频区域进行模拟采样，实现了在 Dyna 中对学习效率提高的有效控制。

Dyna 中基于频率的搜索控制

Frequency-based Search-control in Dyna

Dyna is a fundamental approach to model-based reinforcement learning (MBRL)
that interleaves planning, acting, and learning in an online setting. In the
most typical application of Dyna, the dynamics model is used to generate
one-step transitions from selected start states from the agent's history, which
are used to update the agent's value function or policy as if they were real
experiences. In this work, one-step Dyna was applied to several games from the
Arcade Learning Environment (ALE). We found that the model-based updates
offered surprisingly little benefit over simply performing more updates with
the agent's existing experience, even when using a perfect model. We
hypothesize that to get the most from planning, the model must be used to
generate unfamiliar experience. To test this, we experimented with the "shape"
of planning in multiple different concrete instantiations of Dyna, performing
fewer, longer rollouts, rather than many short rollouts. We found that planning
shape has a profound impact on the efficacy of Dyna for both perfect and
learned models. In addition to these findings regarding Dyna in general, our
results represent, to our knowledge, the first time that a learned dynamics
model has been successfully used for planning in the ALE, suggesting that Dyna
may be a viable approach to MBRL in the ALE and other high-dimensional
problems.

本论文研究了 Dyna（一种基于模型的强化学习）在多个游戏中的应用，发现规划的形式对 Dyna 的有效性有深刻影响，同时也是第一次成功地使用了手头有限的学习动态模型进行规划，该研究表明 Dyna 是解决高维度问题中的模型泛化能力问题的一种可行方法。