The extensive amounts of data required for training deep neural networks pose
significant challenges on storage and transmission fronts. Dataset distillation
has emerged as a promising technique to condense the information of massive
datasets into a much smaller yet representative set of synthetic samples.
However, traditional dataset distillation approaches often struggle to scale
effectively with high-resolution images and more complex architectures due to
the limitations in bi-level optimization. Recently, several works have proposed
exploiting knowledge distillation with decoupled optimization schemes to scale
up dataset distillation. Although these methods effectively address the
scalability issue, they rely on extensive image augmentations requiring the
storage of soft labels for augmented images. In this paper, we introduce
Dataset Distillation using Diffusion Models (D3M) as a novel paradigm for
dataset distillation, leveraging recent advancements in generative
text-to-image foundation models. Our approach utilizes textual inversion, a
technique for fine-tuning text-to-image generative models, to create concise
and informative representations for large datasets. By employing these learned
text prompts, we can efficiently store and infer new samples for introducing
data variability within a fixed memory budget. We show the effectiveness of our
method through extensive experiments across various computer vision benchmark
datasets with different memory budgets.

利用生成式文本到图像模型的最新进展，我们引入了数据集精炼使用扩散模型 (D3M) 作为一种新的数据集精炼范式。通过文本反演技术，我们利用学习到的文本提示为大型数据集创建简洁且有信息量的表示，以在固定的内存预算内有效地存储和推理新样本，并通过在不同内存预算下在各种计算机视觉基准数据集上进行的广泛实验验证了我们方法的有效性。

一个类别一个提示：使用扩散模型进行数据集精炼

One Category One Prompt: Dataset Distillation using Diffusion Models

Spiking Neural Networks (SNNs) offer a promising solution to achieve ultra
low-power/energy computation for solving machine learning tasks. Currently,
most of the SNN architectures are derived from Artificial Neural Networks whose
neurons' architectures and operations are different from SNNs, or developed
without considering memory budgets from the underlying processing hardware.
These limitations hinder the SNNs from reaching their full potential in
accuracy and efficiency. Towards this, we propose SpikeNAS, a novel
memory-aware neural architecture search (NAS) framework for SNNs that can
quickly find an appropriate SNN architecture with high accuracy under the given
memory budgets. To do this, our SpikeNAS employs several key steps: analyzing
the impacts of network operations on the accuracy, enhancing the network
architecture to improve the learning quality, and developing a fast
memory-aware search algorithm. The experimental results show that our SpikeNAS
improves the searching time and maintains high accuracy as compared to
state-of-the-art while meeting the given memory budgets (e.g., 4.4x faster
search with 1.3% accuracy improvement for CIFAR100, using an Nvidia RTX 6000
Ada GPU machine), thereby quickly providing the appropriate SNN architecture
for memory-constrained SNN-based systems.

SpikeNAS 是一种新颖的内存感知神经架构搜索（NAS）框架，用于 SNN，能够在给定的内存预算下快速找到适当的 SNN 架构，提高准确性并满足内存受限的 SNN 系统的需求。

SpikeNAS：一种用于脉冲神经网络系统的快速、内存感知的神经架构搜索框架

SpikeNAS: A Fast Memory-Aware Neural Architecture Search Framework for  Spiking Neural Network Systems

Not forgetting old class knowledge is a key challenge for class-incremental
learning (CIL) when the model continuously adapts to new classes. A common
technique to address this is knowledge distillation (KD), which penalizes
prediction inconsistencies between old and new models. Such prediction is made
with almost new class data, as old class data is extremely scarce due to the
strict memory limitation in CIL. In this paper, we take a deep dive into KD
losses and find that "using new class data for KD" not only hinders the model
adaption (for learning new classes) but also results in low efficiency for
preserving old class knowledge. We address this by "using the placebos of old
classes for KD", where the placebos are chosen from a free image stream, such
as Google Images, in an automatical and economical fashion. To this end, we
train an online placebo selection policy to quickly evaluate the quality of
streaming images (good or bad placebos) and use only good ones for one-time
feed-forward computation of KD. We formulate the policy training process as an
online Markov Decision Process (MDP), and introduce an online learning
algorithm to solve this MDP problem without causing much computation costs. In
experiments, we show that our method 1) is surprisingly effective even when
there is no class overlap between placebos and original old class data, 2) does
not require any additional supervision or memory budget, and 3) significantly
outperforms a number of top-performing CIL methods, in particular when using
lower memory budgets for old class exemplars, e.g., five exemplars per class.

通过在课堂渐进学习中使用替代样本和在线学习算法，我们提出的方法在保留旧课程知识的同时，有效地适应新课程，无需额外的监督或内存预算，并超过其他课堂渐进学习方法的性能。