Adaptive methods are extremely popular in machine learning as they make
learning rate tuning less expensive. This paper introduces a novel optimization
algorithm named KATE, which presents a scale-invariant adaptation of the
well-known AdaGrad algorithm. We prove the scale-invariance of KATE for the
case of Generalized Linear Models. Moreover, for general smooth non-convex
problems, we establish a convergence rate of $O \left(\frac{\log T}{\sqrt{T}}
\right)$ for KATE, matching the best-known ones for AdaGrad and Adam. We also
compare KATE to other state-of-the-art adaptive algorithms Adam and AdaGrad in
numerical experiments with different problems, including complex machine
learning tasks like image classification and text classification on real data.
The results indicate that KATE consistently outperforms AdaGrad and
matches/surpasses the performance of Adam in all considered scenarios.

本研究介绍了一种名为 KATE 的新型优化算法，它是 AdaGrad 算法的一个尺度不变适应版本。我们通过证明其在广义线性模型中具有尺度不变性，并利用数值实验比较 KATE 与 Adam 和 AdaGrad 算法在不同问题上的性能，包括图像分类和文本分类等复杂机器学习任务，结果表明 KATE 在各种情境中始终优于 AdaGrad 并与 Adam 的性能相当甚至超越。

去除平方根：AdaGrad 的新高效的尺度不变版本

Remove that Square Root: A New Efficient Scale-Invariant Version of  AdaGrad

Autoencoders have been successful in learning meaningful representations from
image datasets. However, their performance on text datasets has not been widely
studied. Traditional autoencoders tend to learn possibly trivial
representations of text documents due to their confounding properties such as
high-dimensionality, sparsity and power-law word distributions. In this paper,
we propose a novel k-competitive autoencoder, called KATE, for text documents.
Due to the competition between the neurons in the hidden layer, each neuron
becomes specialized in recognizing specific data patterns, and overall the
model can learn meaningful representations of textual data. A comprehensive set
of experiments show that KATE can learn better representations than traditional
autoencoders including denoising, contractive, variational, and k-sparse
autoencoders. Our model also outperforms deep generative models, probabilistic
topic models, and even word representation models (e.g., Word2Vec) in terms of
several downstream tasks such as document classification, regression, and
retrieval.

这篇论文提出了一种新型 k-competitive 自编码器 KATE，来有效地在文本文档中学习有意义的表示，相较于传统的自编码器和其他模型，KATE 在文本分类、回归、检索等下游任务中表现更好。