Optimization in machine learning, both theoretical and applied, is presently
dominated by first-order gradient methods such as stochastic gradient descent.
Second-order optimization methods, that involve second derivatives and/or
second order statistics of the data, are far less prevalent despite strong
theoretical properties, due to their prohibitive computation, memory and
communication costs. In an attempt to bridge this gap between theoretical and
practical optimization, we present a scalable implementation of a second-order
preconditioned method (concretely, a variant of full-matrix Adagrad), that
along with several critical algorithmic and numerical improvements, provides
significant convergence and wall-clock time improvements compared to
conventional first-order methods on state-of-the-art deep models. Our novel
design effectively utilizes the prevalent heterogeneous hardware architecture
for training deep models, consisting of a multicore CPU coupled with multiple
accelerator units. We demonstrate superior performance compared to
state-of-the-art on very large learning tasks such as machine translation with
Transformers, language modeling with BERT, click-through rate prediction on
Criteo, and image classification on ImageNet with ResNet-50.

本文尝试缩小理论优化与实际优化之间的差距，提出了一种可扩展的二阶预处理方法来优化深度模型，利用异构硬件架构进行训练，相比于常规一阶方法在机器翻译、语言建模、点击率预测和图像分类等任务中表现出优异的性能。

深度学习可扩展的二阶优化

Scalable Second Order Optimization for Deep Learning

We propose the first adversarially robust algorithm for monotone submodular
maximization under single and multiple knapsack constraints with scalable
implementations in distributed and streaming settings. For a single knapsack
constraint, our algorithm outputs a robust summary of almost optimal (up to
polylogarithmic factors) size, from which a constant-factor approximation to
the optimal solution can be constructed. For multiple knapsack constraints, our
approximation is within a constant-factor of the best known non-robust
solution.
We evaluate the performance of our algorithms by comparison to natural
robustifications of existing non-robust algorithms under two objectives: 1)
dominating set for large social network graphs from Facebook and Twitter
collected by the Stanford Network Analysis Project (SNAP), 2) movie
recommendations on a dataset from MovieLens. Experimental results show that our
algorithms give the best objective for a majority of the inputs and show strong
performance even compared to offline algorithms that are given the set of
removals in advance.

该研究提出了针对单个和多个背包约束下的单调次模最大化的首个对抗鲁棒算法，具有可扩展的分布式和流式实现。性能评估结果表明，与现有非鲁棒算法的自然鲁棒化相比，该算法对于大型社交网络图等输入具有最佳的目标结果，并表现出极强的性能，即使与提前给出拆除集合的线下算法相比也是如此。

满足背包约束的敌对鲁棒次模最大化

Adversarially Robust Submodular Maximization under Knapsack Constraints

Hyperbolic embeddings offer excellent quality with few dimensions when
embedding hierarchical data structures like synonym or type hierarchies. Given
a tree, we give a combinatorial construction that embeds the tree in hyperbolic
space with arbitrarily low distortion without using optimization. On WordNet,
our combinatorial embedding obtains a mean-average-precision of 0.989 with only
two dimensions, while Nickel et al.'s recent construction obtains 0.87 using
200 dimensions. We provide upper and lower bounds that allow us to characterize
the precision-dimensionality tradeoff inherent in any hyperbolic embedding. To
embed general metric spaces, we propose a hyperbolic generalization of
multidimensional scaling (h-MDS). We show how to perform exact recovery of
hyperbolic points from distances, provide a perturbation analysis, and give a
recovery result that allows us to reduce dimensionality. The h-MDS approach
offers consistently low distortion even with few dimensions across several
datasets. Finally, we extract lessons from the algorithms and theory above to
design a PyTorch-based implementation that can handle incomplete information
and is scalable.

该研究提出了一种嵌入树形数据结构的超 bolic embeddings 算法，无需优化即可实现任意低失真，同时提供了 h-MDS 方法来嵌入广泛的度量空间以及可处理不完整信息和可扩展的 PyTorch 实现。