An important yet underexplored question in the PAC-Bayes literature is how
much tightness we lose by restricting the posterior family to factorized
Gaussian distributions when optimizing a PAC-Bayes bound. We investigate this
issue by estimating data-independent PAC-Bayes bounds using the optimal
posteriors, comparing them to bounds obtained using MFVI. Concretely, we (1)
sample from the optimal Gibbs posterior using Hamiltonian Monte Carlo, (2)
estimate its KL divergence from the prior with thermodynamic integration, and
(3) propose three methods to obtain high-probability bounds under different
assumptions. Our experiments on the MNIST dataset reveal significant tightness
gaps, as much as 5-6\% in some cases.

该论文通过对 MNIST 数据集进行实验，研究了 PAC-Bayes 参数约束为分解高斯分布时在优化 PAC-Bayes 界限时可能损失的紧密度，结果表明在某些情况下存在 5-6% 的显著紧密度差距。

用哈密尔顿蒙特卡洛估计最优的 PAC-Bayes 界限

Estimating optimal PAC-Bayes bounds with Hamiltonian Monte Carlo

Meta-learning methods typically follow a two-loop framework, where each loop
potentially suffers from notorious overfitting, hindering rapid adaptation and
generalization to new tasks. Existing schemes solve it by enhancing the
mutual-exclusivity or diversity of training samples, but these data
manipulation strategies are data-dependent and insufficiently flexible. This
work alleviates overfitting in meta-learning from the perspective of gradient
regularization and proposes a data-independent \textbf{M}eta-\textbf{G}radient
\textbf{Aug}mentation (\textbf{MGAug}) method. The key idea is to first break
the rote memories by network pruning to address memorization overfitting in the
inner loop, and then the gradients of pruned sub-networks naturally form the
high-quality augmentation of the meta-gradient to alleviate learner overfitting
in the outer loop. Specifically, we explore three pruning strategies, including
\textit{random width pruning}, \textit{random parameter pruning}, and a newly
proposed \textit{catfish pruning} that measures a Meta-Memorization Carrying
Amount (MMCA) score for each parameter and prunes high-score ones to break rote
memories as much as possible. The proposed MGAug is theoretically guaranteed by
the generalization bound from the PAC-Bayes framework. In addition, we extend a
lightweight version, called MGAug-MaxUp, as a trade-off between performance
gains and resource overhead. Extensive experiments on multiple few-shot
learning benchmarks validate MGAug's effectiveness and significant improvement
over various meta-baselines. The code is publicly available at
https://github.com/xxLifeLover/Meta-Gradient-Augmentation.

本文从梯度规范化的角度出发，提出了一种数据无关的元梯度增强 (MGAug) 方法，通过网络修剪来打破背诵性记忆，并探讨了三种修剪策略，实验表明该方法在多个 few-shot 学习基准测试中显著提高了性能

通过元梯度增强改善元学习的泛化能力

Improving Generalization in Meta-Learning via Meta-Gradient Augmentation

Model compression is eminently suited for deploying deep learning on
IoT-devices. However, existing model compression techniques rely on access to
the original or some alternate dataset. In this paper, we address the model
compression problem when no real data is available, e.g., when data is private.
To this end, we propose Dream Distillation, a data-independent model
compression framework. Our experiments show that Dream Distillation can achieve
88.5% accuracy on the CIFAR-10 test set without actually training on the
original data!

本文提出了一种基于 Dream Distillation 的数据无关的模型压缩框架，能够在没有数据训练的情况下，在 CIFAR-10 测试集上达到 88.5% 的精度。