Informative gradients are often lost in large batch updates. We propose a robust mechanism to reinforce the sparse components within a random batch of data points. A finite queue of online gradients is used to determine their expected instantaneous statistics. We propose a function to measure the scarcity of incoming gradients using these statistics and establish the theoretical ground of this mechanism. To minimize conflicting components within large mini-batches, samples are grouped with aligned objectives by clustering based on inherent feature space. Sparsity is measured for each centroid and weighted accordingly. A strong intuitive criterion to squeeze out redundant information from each cluster is the backbone of the system. It makes rare information indifferent to aggressive momentum also exhibits superior performance with larger mini-batch horizon. The effective length of the queue kept variable to follow the local loss pattern. The contribution of our method is to restore intra-mini-batch diversity at the same time widening the optimal batch boundary. Both of these collectively drive it deeper towards the minima. Our method has shown superior performance for CIFAR10, MNIST, and Reuters News category dataset compared to mini-batch gradient descent.

使用在线梯度队列和聚类的机制来增强数据批次中的稀疏成分，提高大型批次更新中信息梯度的效果。通过测量梯度稀缺程度和聚类方法，该机制在最小化冲突成分的同时，增加了批次边界的优化，从而更好地接近最小值。该方法在 CIFAR10、MNIST 和 Reuters News 分类数据集上表现出了优越性能，相较于小批量梯度下降方法。

Grad Queue: 强化稀疏梯度的概率框架