User-generated social media data is constantly changing as new trends
influence online discussion, causing distribution shift in test data for social
media NLP applications. In addition, training data is often subject to change
as user data is deleted. Most current NLP systems are static and rely on fixed
training data. As a result, they are unable to adapt to temporal change -- both
test distribution shift and deleted training data -- without frequent, costly
re-training. In this paper, we study temporal adaptation through the task of
longitudinal hashtag prediction and propose a non-parametric technique as a
simple but effective solution: non-parametric classifiers use datastores which
can be updated, either to adapt to test distribution shift or training data
deletion, without re-training. We release a new benchmark dataset comprised of
7.13M Tweets from 2021, along with their hashtags, broken into consecutive
temporal buckets. We compare parametric neural hashtag classification and
hashtag generation models, which need re-training for adaptation, with a
non-parametric, training-free dense retrieval method that returns the nearest
neighbor's hashtags based on text embedding distance. In experiments on our
longitudinal Twitter dataset we find that dense nearest neighbor retrieval has
a relative performance gain of 64.12% over the best parametric baseline on test
sets that exhibit distribution shift without requiring gradient-based
re-training. Furthermore, we show that our datastore approach is particularly
well-suited to dynamically deleted user data, with negligible computational
cost and performance loss. Our novel benchmark dataset and empirical analysis
can support future inquiry into the important challenges presented by
temporality in the deployment of AI systems on real-world user data.

本文针对用户生成的社交媒体数据不断变化，现有自然语言处理系统无法适应且需要大量重新训练的问题，提出了一种基于非参数分类器的简单有效的时间适应方法，并通过预测长期 Hashtag 变化的任务进行了实验。结果表明，非参数分类器的分类效果相比最佳参数基线提高了 64.12％，而且被动删除用户数据时也表现出极高的性能。

社交媒體話題分類的非參數時間適應

Non-Parametric Temporal Adaptation for Social Media Topic Classification

Training convolutional networks (CNN's) that fit on a single GPU with
minibatch stochastic gradient descent has become effective in practice.
However, there is still no effective method for training large CNN's that do
not fit in the memory of a few GPU cards, or for parallelizing CNN training. In
this work we show that a simple hard mixture of experts model can be
efficiently trained to good effect on large scale hashtag (multilabel)
prediction tasks. Mixture of experts models are not new (Jacobs et. al. 1991,
Collobert et. al. 2003), but in the past, researchers have had to devise
sophisticated methods to deal with data fragmentation. We show empirically that
modern weakly supervised data sets are large enough to support naive
partitioning schemes where each data point is assigned to a single expert.
Because the experts are independent, training them in parallel is easy, and
evaluation is cheap for the size of the model. Furthermore, we show that we can
use a single decoding layer for all the experts, allowing a unified feature
embedding space. We demonstrate that it is feasible (and in fact relatively
painless) to train far larger models than could be practically trained with
standard CNN architectures, and that the extra capacity can be well used on
current datasets.

本研究提出了一种简单的 Mixture of Experts 模型，在大规模、多标签的预测任务中表现良好，适用于数据集分布不均、单个 GPU 存储不下的情形，并支持并行训练和统一的特征嵌入空间。该模型的表现表明可以用来训练更大的深度学习模型，拥有更强的处理能力。