Feature attribution explains neural network outputs by identifying relevant
input features. How do we know if the identified features are indeed relevant
to the network? This notion is referred to as faithfulness, an essential
property that reflects the alignment between the identified (attributed)
features and the features used by the model. One recent trend to test
faithfulness is to design the data such that we know which input features are
relevant to the label and then train a model on the designed data.
Subsequently, the identified features are evaluated by comparing them with
these designed ground truth features. However, this idea has the underlying
assumption that the neural network learns to use all and only these designed
features, while there is no guarantee that the learning process trains the
network in this way. In this paper, we solve this missing link by explicitly
designing the neural network by manually setting its weights, along with
designing data, so we know precisely which input features in the dataset are
relevant to the designed network. Thus, we can test faithfulness in
AttributionLab, our designed synthetic environment, which serves as a sanity
check and is effective in filtering out attribution methods. If an attribution
method is not faithful in a simple controlled environment, it can be unreliable
in more complex scenarios. Furthermore, the AttributionLab environment serves
as a laboratory for controlled experiments through which we can study feature
attribution methods, identify issues, and suggest potential improvements.

设计 AttributionLab 环境用于测试特征归因方法的可靠性，并提供了一个控制实验室，用于研究特征归因方法、识别问题并提出潜在改进。

AttributionLab：在可控环境下的特征归因的可信度

AttributionLab: Faithfulness of Feature Attribution Under Controllable  Environments

Pretraining is the preliminary and fundamental step in developing capable
language models (LM). Despite this, pretraining data design is critically
under-documented and often guided by empirically unsupported intuitions. To
address this, we pretrain 28 1.5B parameter decoder-only models, training on
data curated (1) at different times, (2) with varying toxicity and quality
filters, and (3) with different domain compositions. First, we quantify the
effect of pretraining data age. A temporal shift between evaluation data and
pretraining data leads to performance degradation, which is not overcome by
finetuning. Second, we explore the effect of quality and toxicity filters,
showing a trade-off between performance on standard benchmarks and risk of
toxic generations. Our findings indicate there does not exist a
one-size-fits-all solution to filtering training data. We also find that the
effects of different types of filtering are not predictable from text domain
characteristics. Lastly, we empirically validate that the inclusion of
heterogeneous data sources, like books and web, is broadly beneficial and
warrants greater prioritization. These findings constitute the largest set of
experiments to validate, quantify, and expose many undocumented intuitions
about text pretraining, which we hope will help support more informed
data-centric decisions in LM development.

对预训练数据合理性进行了研究，为语言模型开发做出了数据方面的决策提供支持。