The poor cross-architecture generalization of dataset distillation greatly
weakens its practical significance. This paper attempts to mitigate this issue
through an empirical study, which suggests that the synthetic datasets undergo
an inductive bias towards the distillation model. Therefore, the evaluation
model is strictly confined to having similar architectures of the distillation
model. We propose a novel method of EvaLuation with distillation Feature (ELF),
which utilizes features from intermediate layers of the distillation model for
the cross-architecture evaluation. In this manner, the evaluation model learns
from bias-free knowledge therefore its architecture becomes unfettered while
retaining performance. By performing extensive experiments, we successfully
prove that ELF can well enhance the cross-architecture generalization of
current DD methods. Code of this project is at
https://github.com/Lirui-Zhao/ELF.

通过实证研究，本文提出了一种新颖的评估方法 ELF，通过利用蒸馏模型的中间层特征进行跨架构评估，从无偏重的知识中学习，以提高当前 DD 方法的跨架构泛化能力。

通过实证研究提升数据集蒸馏的跨架构泛化能力

Boosting the Cross-Architecture Generalization of Dataset Distillation  through an Empirical Study

The development and spread of information and communication technologies have
increased and diversified information. However, the increase in the volume and
the selection of information does not necessarily promote understanding. In
addition, conventional evaluations of information transfer have focused only on
the arrival of information to the receivers. They need to sufficiently take
into account the receivers' understanding of the information after it has been
acquired, which is the original purpose of the evaluation. In this study, we
propose the concept of "information digestion," which refers to the receivers'
correct understanding of the acquired information, its contents, and its
purpose. In the experiment, we proposed an evaluation model of information
digestibility using hierarchical factor analysis and extracted factors that
constitute digestibility by four types of media.

本研究提出了 “信息消化” 的概念，并设计了一种评估模型，使用分层因子分析提取了构成消化能力的因子。

文图信息传输消化效率的成分因子提取

Extraction of Constituent Factors of Digestion Efficiency in Information Transfer by Media Composed of Texts and Images

Deliberation is a common and natural behavior in human daily life. For
example, when writing papers or articles, we usually first write drafts, and
then iteratively polish them until satisfied. In light of such a human
cognitive process, we propose DECOM, which is a multi-pass deliberation
framework for automatic comment generation. DECOM consists of multiple
Deliberation Models and one Evaluation Model. Given a code snippet, we first
extract keywords from the code and retrieve a similar code fragment from a
pre-defined corpus. Then, we treat the comment of the retrieved code as the
initial draft and input it with the code and keywords into DECOM to start the
iterative deliberation process. At each deliberation, the deliberation model
polishes the draft and generates a new comment. The evaluation model measures
the quality of the newly generated comment to determine whether to end the
iterative process or not. When the iterative process is terminated, the
best-generated comment will be selected as the target comment. Our approach is
evaluated on two real-world datasets in Java (87K) and Python (108K), and
experiment results show that our approach outperforms the state-of-the-art
baselines. A human evaluation study also confirms the comments generated by
DECOM tend to be more readable, informative, and useful.

DECOM 是一种多通道决策框架，用于自动生成代码评论，并通过评估模型检测评论的质量，实验结果证明 DECOM 产生的评论更可读、信息量更大和更有用。

通过多次深入讨论实现自动评论生成

Automatic Comment Generation via Multi-Pass Deliberation

Progress in machine learning is measured by careful evaluation on problems of
outstanding common interest. However, the proliferation of benchmark suites and
environments, adversarial attacks, and other complications has diluted the
basic evaluation model by overwhelming researchers with choices. Deliberate or
accidental cherry picking is increasingly likely, and designing well-balanced
evaluation suites requires increasing effort. In this paper we take a step back
and propose Nash averaging. The approach builds on a detailed analysis of the
algebraic structure of evaluation in two basic scenarios: agent-vs-agent and
agent-vs-task. The key strength of Nash averaging is that it automatically
adapts to redundancies in evaluation data, so that results are not biased by
the incorporation of easy tasks or weak agents. Nash averaging thus encourages
maximally inclusive evaluation -- since there is no harm (computational cost
aside) from including all available tasks and agents.

本文介绍了一种叫作 Nash 平均的评估方法，能够自动适应评估数据中的冗余信息，从而避免了采用简单任务或弱智能体造成的结果偏差，实现了最大程度的评估包容性。