Motivated by the goals of dataset pruning and defect identification, a
growing body of methods have been developed to score individual examples within
a dataset. These methods, which we call "example difficulty scores", are
typically used to rank or categorize examples, but the consistency of rankings
between different training runs, scoring methods, and model architectures is
generally unknown. To determine how example rankings vary due to these random
and controlled effects, we systematically compare different formulations of
scores over a range of runs and model architectures. We find that scores
largely share the following traits: they are noisy over individual runs of a
model, strongly correlated with a single notion of difficulty, and reveal
examples that range from being highly sensitive to insensitive to the inductive
biases of certain model architectures. Drawing from statistical genetics, we
develop a simple method for fingerprinting model architectures using a few
sensitive examples. These findings guide practitioners in maximizing the
consistency of their scores (e.g. by choosing appropriate scoring methods,
number of runs, and subsets of examples), and establishes comprehensive
baselines for evaluating scores in the future.

通过对随机和受控因素的比较，我们发现例子排名在不同的训练运行、评分方法和模型架构间存在很大的一致性。同时，我们还开发了一种简单的方法来使用一些敏感的例子对模型架构进行指纹识别。这些发现指导从业者在最大化评分一致性方面（例如通过选择适当的评分方法、运行次数和例子子集）并为将来评估评分建立全面的基准。

数据集难度与归纳偏差的作用

Dataset Difficulty and the Role of Inductive Bias

This work provides a solution to the challenge of small amounts of training
data in Non-Destructive Ultrasonic Testing for composite components. It was
demonstrated that direct simulation alone is ineffective at producing training
data that was representative of the experimental domain due to poor noise
reconstruction. Therefore, four unique synthetic data generation methods were
proposed which use semi-analytical simulated data as a foundation. Each method
was evaluated on its classification performance of real experimental images
when trained on a Convolutional Neural Network which underwent hyperparameter
optimization using a genetic algorithm. The first method introduced task
specific modifications to CycleGAN, to learn the mapping from physics-based
simulations of defect indications to experimental indications in resulting
ultrasound images. The second method was based on combining real experimental
defect free images with simulated defect responses. The final two methods fully
simulated the noise responses at an image and signal level respectively. The
purely simulated data produced a mean classification F1 score of 0.394.
However, when trained on the new synthetic datasets, a significant improvement
in classification performance on experimental data was realized, with mean
classification F1 scores of 0.843, 0.688, 0.629, and 0.738 for the respective
approaches.

本文提出了四种唯一的数据生成方法解决了在复合材料组件的无损超声波检测过程中由于训练数据数量不足而导致的挑战，使用卷积神经网络进行训练，并在新的合成数据集上进行训练，取得了较好的实验数据分类效果。