Learning representations of stochastic processes is an emerging problem in
machine learning with applications from meta-learning to physical object models
to time series. Typical methods rely on exact reconstruction of observations,
but this approach breaks down as observations become high-dimensional or noise
distributions become complex. To address this, we propose a unifying framework
for learning contrastive representations of stochastic processes (CReSP) that
does away with exact reconstruction. We dissect potential use cases for
stochastic process representations, and propose methods that accommodate each.
Empirically, we show that our methods are effective for learning
representations of periodic functions, 3D objects and dynamical processes. Our
methods tolerate noisy high-dimensional observations better than traditional
approaches, and the learned representations transfer to a range of downstream
tasks.

本论文提出了一种统一的学习随机过程对比表示的方法，称为 CReSP，从而弥补了精确重构观察数据的方法在高维度或噪声分布复杂时不可靠的缺陷，并在周期函数、三维物体和动态过程的表示学习等方面取得了显著效果，并且该方法对高维度噪声观测的容忍度比传统方法更高，并且所学表示可以用于各种下游任务。

随机过程的对比表示

On Contrastive Representations of Stochastic Processes

Previous literature offers limited clues on how to learn a periodic function
using modern neural networks. We start with a study of the extrapolation
properties of neural networks; we prove and demonstrate experimentally that the
standard activations functions, such as ReLU, tanh, sigmoid, along with their
variants, all fail to learn to extrapolate simple periodic functions. We
hypothesize that this is due to their lack of a "periodic" inductive bias. As a
fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$,
which achieves the desired periodic inductive bias to learn a periodic function
while maintaining a favorable optimization property of the ReLU-based
activations. Experimentally, we apply the proposed method to temperature and
financial data prediction.

使用新型激活函数 $x + sin^2 (x)$，结合传统 ReLU 激活函数的优势，来解决传统激活函数（如 ReLU、tanh、sigmoid 等）在学习周期函数方面的普遍失败。经实验证明，该方法能够成功地预测温度和金融数据。