We study gradient flow on the exponential loss for a classification problem
with a one-layer softmax attention model, where the key and query weight
matrices are trained separately. Under a separability assumption on the data,
we show that when gradient flow achieves the minimal loss value, it further
implicitly minimizes the nuclear norm of the product of the key and query
weight matrices. Such implicit regularization can be described by a Support
Vector Machine (SVM) problem with respect to the attention weights. This
finding contrasts with prior results showing that the gradient descent induces
an implicit regularization on the Frobenius norm on the product weight matrix
when the key and query matrices are combined into a single weight matrix for
training. For diagonal key and query matrices, our analysis builds upon the
reparameterization technique and exploits approximate KKT conditions of the SVM
associated with the classification data. Moreover, the results are extended to
general weights configurations given proper alignment of the weight matrices'
singular spaces with the data features at initialization.

研究一层 softmax 注意模型上指数损失的梯度流问题，通过分离地训练键值矩阵，我们在数据可分性假设下证明梯度流在达到最小损失值时，进一步隐式地最小化键值矩阵乘积的核范数，这种隐式正则化可以用关注权重的支持向量机（SVM）问题来描述。与先前结果相反，当将键值矩阵合并为单个权重矩阵进行训练时，梯度下降对于乘积权重矩阵的 Frobenius 范数引入了隐式正则化。对于对角键值和查询矩阵，我们的分析建立在重新参数化技术和利用与分类数据相关的 SVM 近似 KKT 条件的基础上。此外，该结果对于给定适当对齐的权重矩阵奇异空间与数据特征的初始化情况也同样适用。