Large-scale sequence modeling has sparked rapid advances that now extend into
biology and genomics. However, modeling genomic sequences introduces challenges
such as the need to model long-range token interactions, the effects of
upstream and downstream regions of the genome, and the reverse complementarity
(RC) of DNA. Here, we propose an architecture motivated by these challenges
that builds off the long-range Mamba block, and extends it to a BiMamba
component that supports bi-directionality, and to a MambaDNA block that
additionally supports RC equivariance. We use MambaDNA as the basis of
Caduceus, the first family of RC equivariant bi-directional long-range DNA
language models, and we introduce pre-training and fine-tuning strategies that
yield Caduceus DNA foundation models. Caduceus outperforms previous long-range
models on downstream benchmarks; on a challenging long-range variant effect
prediction task, Caduceus exceeds the performance of 10x larger models that do
not leverage bi-directionality or equivariance.

提出了一种基于长程 Mamba 块构建的双向长程 DNA 语言模型，命名为 Caduceus，它在下游基准测试中表现优于之前的长程模型，并在具有挑战性的长程变体效应预测任务中超越不利用双向性或等变性的 10 倍更大的模型。

双向对称长距离 DNA 序列建模

Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling

Improving the explainability of the results from machine learning methods has
become an important research goal. Here, we study the problem of making
clusters more interpretable by extending a recent approach of [Davidson et al.,
NeurIPS 2018] for constructing succinct representations for clusters. Given a
set of objects $S$, a partition $\pi$ of $S$ (into clusters), and a universe
$T$ of tags such that each element in $S$ is associated with a subset of tags,
the goal is to find a representative set of tags for each cluster such that
those sets are pairwise-disjoint and the total size of all the representatives
is minimized. Since this problem is NP-hard in general, we develop
approximation algorithms with provable performance guarantees for the problem.
We also show applications to explain clusters from datasets, including clusters
of genomic sequences that represent different threat levels.

本文研究如何提高机器学习方法结果的可解释性，探讨了通过构建对聚类进行简洁表示的方法，提出了可证明性能保证的近似算法，并应用于基因组序列的不同威胁级别的聚类解释。

用于可解释性的生成可证明近似最优的聚类描述符的高效算法

Efficient Algorithms for Generating Provably Near-Optimal Cluster  Descriptors for Explainability

Discriminative neural networks offer little or no performance guarantees when
deployed on data not generated by the same process as the training
distribution. On such out-of-distribution (OOD) inputs, the prediction may not
only be erroneous, but confidently so, limiting the safe deployment of
classifiers in real-world applications. One such challenging application is
bacteria identification based on genomic sequences, which holds the promise of
early detection of diseases, but requires a model that can output low
confidence predictions on OOD genomic sequences from new bacteria that were not
present in the training data. We introduce a genomics dataset for OOD detection
that allows other researchers to benchmark progress on this important problem.
We investigate deep generative model based approaches for OOD detection and
observe that the likelihood score is heavily affected by population level
background statistics. We propose a likelihood ratio method for deep generative
models which effectively corrects for these confounding background statistics.
We benchmark the OOD detection performance of the proposed method against
existing approaches on the genomics dataset and show that our method achieves
state-of-the-art performance. We demonstrate the generality of the proposed
method by showing that it significantly improves OOD detection when applied to
deep generative models of images.

研究讨论了关于基于基因组序列的细菌识别的 out-of-distribution (OOD) inputs 问题，提出了用于深度生成模型的似然比方法来纠正背景统计数据偏差，并在 genomics 数据集上进行了实验。结果表明，我们的方法在 OOD 检测方面取得了最好的性能，并且我们证明了该方法的广泛适用性。

用于识别样本分布外点的似然比

Likelihood Ratios for Out-of-Distribution Detection

This paper applies a deep convolutional/highway MLP framework to classify
genomic sequences on the transcription factor binding site task. To make the
model understandable, we propose an optimization driven strategy to extract
"motifs", or symbolic patterns which visualize the positive class learned by
the network. We show that our system, Deep Motif (DeMo), extracts motifs that
are similar to, and in some cases outperform the current well known motifs. In
addition, we find that a deeper model consisting of multiple convolutional and
highway layers can outperform a single convolutional and fully connected layer
in the previous state-of-the-art.

本文利用深层卷积 / MLP 框架对基因组序列进行分类，在优化驱动策略的指导下提取 “motifs” 以可视化网络学习的正向分类。实验证明，本文提出的 Deep Motif（DeMo）提取的 motifs 类似于并在某些情况下优于当前已知的 motifs；此外，本文还发现由多个卷积和 highway 层组成的深度模型可以胜过以前最先进的单个卷积和全连接层。