Language exhibits structure at different scales, ranging from subwords to
words, sentences, paragraphs, and documents. To what extent do deep models
capture information at these scales, and can we force them to better capture
structure across this hierarchy? We approach this question by focusing on
individual neurons, analyzing the behavior of their activations at different
timescales. We show that signal processing provides a natural framework for
separating structure across scales, enabling us to 1) disentangle
scale-specific information in existing embeddings and 2) train models to learn
more about particular scales. Concretely, we apply spectral filters to the
activations of a neuron across an input, producing filtered embeddings that
perform well on part of speech tagging (word-level), dialog speech acts
classification (utterance-level), or topic classification (document-level),
while performing poorly on the other tasks. We also present a prism layer for
training models, which uses spectral filters to constrain different neurons to
model structure at different scales. Our proposed BERT + Prism model can better
predict masked tokens using long-range context and produces multiscale
representations that perform better at utterance- and document-level tasks. Our
methods are general and readily applicable to other domains besides language,
such as images, audio, and video.

该研究通过分析神经元在不同时间尺度下的激活行为，提出了一种自然的信号处理框架来分离不同尺度上的结构，并应用谱滤波器训练模型在不同尺度上进行学习，产生了更好的预测性能。

透过棱镜看语言：一种用于多尺度语言表示的光谱方法

Language Through a Prism: A Spectral Approach for Multiscale Language  Representations

The success of deep convolutional architectures is often attributed in part
to their ability to learn multiscale and invariant representations of natural
signals. However, a precise study of these properties and how they affect
learning guarantees is still missing. In this paper, we consider deep
convolutional representations of signals; we study their invariance to
translations and to more general groups of transformations, their stability to
the action of diffeomorphisms, and their ability to preserve signal
information. This analysis is carried by introducing a multilayer kernel based
on convolutional kernel networks and by studying the geometry induced by the
kernel mapping. We then characterize the corresponding reproducing kernel
Hilbert space (RKHS), showing that it contains a large class of convolutional
neural networks with homogeneous activation functions. This analysis allows us
to separate data representation from learning, and to provide a canonical
measure of model complexity, the RKHS norm, which controls both stability and
generalization of any learned model. In addition to models in the constructed
RKHS, our stability analysis also applies to convolutional networks with
generic activations such as rectified linear units, and we discuss its
relationship with recent generalization bounds based on spectral norms.

本文研究了深度卷积体系结构的多尺度不变表示问题，提出了基于卷积核网络的多层核方法，分析了核映射引起的几何学效应，表明可以将数据表示与学习分离，提出了模型复杂度的规范化测量，即控制所学模型的稳定性和泛化能力的重复核希尔伯特空间规范，证明了已有卷积神经网络能够映射到该空间中。

深度卷积表达的群不变性、形变稳定性和复杂性

Group Invariance, Stability to Deformations, and Complexity of Deep  Convolutional Representations

We present Walklets, a novel approach for learning multiscale representations
of vertices in a network. In contrast to previous works, these representations
explicitly encode multiscale vertex relationships in a way that is analytically
derivable.
Walklets generates these multiscale relationships by subsampling short random
walks on the vertices of a graph. By `skipping' over steps in each random walk,
our method generates a corpus of vertex pairs which are reachable via paths of
a fixed length. This corpus can then be used to learn a series of latent
representations, each of which captures successively higher order relationships
from the adjacency matrix.
We demonstrate the efficacy of Walklets's latent representations on several
multi-label network classification tasks for social networks such as
BlogCatalog, DBLP, Flickr, and YouTube. Our results show that Walklets
outperforms new methods based on neural matrix factorization. Specifically, we
outperform DeepWalk by up to 10% and LINE by 58% Micro-F1 on challenging
multi-label classification tasks. Finally, Walklets is an online algorithm, and
can easily scale to graphs with millions of vertices and edges.

我们提出了一种名为 Walklets 的算法，通过子采样短的随机游走，在图的顶点之间建立多尺度关系，从而生成多维表示，用于社交网络等多标签网络分类任务，其效果优于 DeepWalk 和 LINE 等方法，且为在线算法，可轻松扩展到数百万个顶点和边。