Continuous-depth learning has recently emerged as a novel perspective on deep
learning, improving performance in tasks related to dynamical systems and
density estimation. Core to these approaches is the neural differential
equation, whose forward passes are the solutions of an initial value problem
parametrized by a neural network. Unlocking the full potential of
continuous-depth models requires a different set of software tools, due to
peculiar differences compared to standard discrete neural networks, e.g
inference must be carried out via numerical solvers. We introduce TorchDyn, a
PyTorch library dedicated to continuous-depth learning, designed to elevate
neural differential equations to be as accessible as regular plug-and-play deep
learning primitives. This objective is achieved by identifying and subdividing
different variants into common essential components, which can be combined and
freely repurposed to obtain complex compositional architectures. TorchDyn
further offers step-by-step tutorials and benchmarks designed to guide
researchers and contributors.

本文介绍了 TorchDyn，一个专为连续深度学习而设计的 PyTorch 库，旨在将神经微分方程提升为与常规深度学习元素一样易用，该库基于将不同变量标识和子分成通用基本组件，可以自由地重新组合以获得复杂的组合架构，同时 TorchDyn 还提供逐步教程和基准测试，旨在引导研究人员和贡献者。

TorchDyn: 神经微分方程库

TorchDyn: A Neural Differential Equations Library

We consider the problem of learning general-purpose, paraphrastic sentence
embeddings based on supervision from the Paraphrase Database (Ganitkevitch et
al., 2013). We compare six compositional architectures, evaluating them on
annotated textual similarity datasets drawn both from the same distribution as
the training data and from a wide range of other domains. We find that the most
complex architectures, such as long short-term memory (LSTM) recurrent neural
networks, perform best on the in-domain data. However, in out-of-domain
scenarios, simple architectures such as word averaging vastly outperform LSTMs.
Our simplest averaging model is even competitive with systems tuned for the
particular tasks while also being extremely efficient and easy to use.
In order to better understand how these architectures compare, we conduct
further experiments on three supervised NLP tasks: sentence similarity,
entailment, and sentiment classification. We again find that the word averaging
models perform well for sentence similarity and entailment, outperforming
LSTMs. However, on sentiment classification, we find that the LSTM performs
very strongly-even recording new state-of-the-art performance on the Stanford
Sentiment Treebank.
We then demonstrate how to combine our pretrained sentence embeddings with
these supervised tasks, using them both as a prior and as a black box feature
extractor. This leads to performance rivaling the state of the art on the SICK
similarity and entailment tasks. We release all of our resources to the
research community with the hope that they can serve as the new baseline for
further work on universal sentence embeddings.

研究使用 Paraphrase Database 监督学习通用的、同义句的句子嵌入，比较 6 种组成结构并在相似度数据集和三个 NLP 任务上进行评估，在领域内复杂的结构（如 LSTM）有更好的性能，但在领域外使用简单架构如词平均比 LSTM 表现更好，同时演示如何将预训练的句子嵌入与监督任务相结合。