Dataset Distillation (DD) aims to distill knowledge from extensive datasets
into more compact ones while preserving performance on the test set, thereby
reducing storage costs and training expenses. However, existing methods often
suffer from computational intensity, particularly exhibiting suboptimal
performance with large dataset sizes due to the lack of a robust theoretical
framework for analyzing the DD problem. To address these challenges, we propose
the BAyesian optimal CONdensation framework (BACON), which is the first work to
introduce the Bayesian theoretical framework to the literature of DD. This
framework provides theoretical support for enhancing the performance of DD.
Furthermore, BACON formulates the DD problem as the minimization of the
expected risk function in joint probability distributions using the Bayesian
framework. Additionally, by analyzing the expected risk function for optimal
condensation, we derive a numerically feasible lower bound based on specific
assumptions, providing an approximate solution for BACON. We validate BACON
across several datasets, demonstrating its superior performance compared to
existing state-of-the-art methods. For instance, under the IPC-10 setting,
BACON achieves a 3.46% accuracy gain over the IDM method on the CIFAR-10
dataset and a 3.10% gain on the TinyImageNet dataset. Our extensive experiments
confirm the effectiveness of BACON and its seamless integration with existing
methods, thereby enhancing their performance for the DD task. Code and
distilled datasets are available at BACON.

通过引入贝叶斯理论框架，提出了 BAyesian optimal CONdensation framework (BACON)，该框架提供了支持，用于提高数据集蒸馏的性能。验证了 BACON 在多个数据集上的优越性能，并证实了其与现有方法的无缝集成，从而提升了数据集蒸馏任务的性能。

BACON：基于贝叶斯优化的数据集精简框架

BACON: Bayesian Optimal Condensation Framework for Dataset Distillation

Dataset distillation extracts a small set of synthetic training samples from
a large dataset with the goal of achieving competitive performance on test data
when trained on this sample. In this work, we tackle dataset distillation at
its core by treating it directly as a bilevel optimization problem.
Re-examining the foundational back-propagation through time method, we study
the pronounced variance in the gradients, computational burden, and long-term
dependencies. We introduce an improved method: Random Truncated Backpropagation
Through Time (RaT-BPTT) to address them. RaT-BPTT incorporates a truncation
coupled with a random window, effectively stabilizing the gradients and
speeding up the optimization while covering long dependencies. This allows us
to establish new state-of-the-art for a variety of standard dataset benchmarks.
A deeper dive into the nature of distilled data unveils pronounced
intercorrelation. In particular, subsets of distilled datasets tend to exhibit
much worse performance than directly distilled smaller datasets of the same
size. Leveraging RaT-BPTT, we devise a boosting mechanism that generates
distilled datasets that contain subsets with near optimal performance across
different data budgets.

通过 RaT-BPTT 方法解决数据集精炼的核心问题，从而建立起一种新的数据集精炼技术，用于生成包含近乎最佳性能子集的精炼数据集。