The paradigm of machine intelligence moves from purely supervised learning to
a more practical scenario when many loosely related unlabeled data are
available and labeled data is scarce. Most existing algorithms assume that the
underlying task distribution is stationary. Here we consider a more realistic
and challenging setting in that task distributions evolve over time. We name
this problem as Semi-supervised meta-learning with Evolving Task diStributions,
abbreviated as SETS. Two key challenges arise in this more realistic setting:
(i) how to use unlabeled data in the presence of a large amount of unlabeled
out-of-distribution (OOD) data; and (ii) how to prevent catastrophic forgetting
on previously learned task distributions due to the task distribution shift. We
propose an OOD Robust and knowleDge presErved semi-supeRvised meta-learning
approach (ORDER), to tackle these two major challenges. Specifically, our ORDER
introduces a novel mutual information regularization to robustify the model
with unlabeled OOD data and adopts an optimal transport regularization to
remember previously learned knowledge in feature space. In addition, we test
our method on a very challenging dataset: SETS on large-scale non-stationary
semi-supervised task distributions consisting of (at least) 72K tasks. With
extensive experiments, we demonstrate the proposed ORDER alleviates forgetting
on evolving task distributions and is more robust to OOD data than related
strong baselines.

本文提出了一种半监督元学习方法（SETS）来解决不断演化的任务分布和大量的未标记数据存在时的元学习问题，并通过引入相互信息正则化来稳健地利用未标记外部分布数据以及采用最优传输正则化来在特征空间记住已学习的知识。在大规模非稳态半监督任务分布数据集上的测试中，实验证明了本文方法相对于相关强基线具有更好的鲁棒性。

大规模非静态任务分布下的低遗忘元学习

Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task  Distributions

This paper studies defense mechanisms against model inversion (MI) attacks --
a type of privacy attacks aimed at inferring information about the training
data distribution given the access to a target machine learning model. Existing
defense mechanisms rely on model-specific heuristics or noise injection. While
being able to mitigate attacks, existing methods significantly hinder model
performance. There remains a question of how to design a defense mechanism that
is applicable to a variety of models and achieves better utility-privacy
tradeoff. In this paper, we propose the Mutual Information Regularization based
Defense (MID) against MI attacks. The key idea is to limit the information
about the model input contained in the prediction, thereby limiting the ability
of an adversary to infer the private training attributes from the model
prediction. Our defense principle is model-agnostic and we present tractable
approximations to the regularizer for linear regression, decision trees, and
neural networks, which have been successfully attacked by prior work if not
attached with any defenses. We present a formal study of MI attacks by devising
a rigorous game-based definition and quantifying the associated information
leakage. Our theoretical analysis sheds light on the inefficacy of DP in
defending against MI attacks, which has been empirically observed in several
prior works. Our experiments demonstrate that MID leads to state-of-the-art
performance for a variety of MI attacks, target models and datasets.

该研究提出了一种基于互信息正则化的防御模型反演攻击的方法，通过限制模型输入中包含的信息，从而限制对手从模型预测中推断出私有属性的能力。此方法能够应用于多种模型并在保护隐私和工具合理性方面取得更好的平衡。