A common way to evaluate a dataset in ML involves training a model on this
dataset and assessing the model's performance on a test set. However, this
approach has two issues: (1) it may incentivize undesirable data manipulation
in data marketplaces, as the self-interested data providers seek to modify the
dataset to maximize their evaluation scores; (2) it may select datasets that
overfit to potentially small test sets. We propose a new data valuation method
that provably guarantees the following: data providers always maximize their
expected score by truthfully reporting their observed data. Any manipulation of
the data, including but not limited to data duplication, adding random data,
data removal, or re-weighting data from different groups, cannot increase their
expected score. Our method, following the paradigm of proper scoring rules,
measures the pointwise mutual information (PMI) of the test dataset and the
evaluated dataset. However, computing the PMI of two datasets is challenging.
We introduce a novel PMI measuring method that greatly improves tractability
within Bayesian machine learning contexts. This is accomplished through a new
characterization of PMI that relies solely on the posterior probabilities of
the model parameter at an arbitrarily selected value. Finally, we support our
theoretical results with simulations and further test the effectiveness of our
data valuation method in identifying the top datasets among multiple data
providers. Interestingly, our method outperforms the standard approach of
selecting datasets based on the trained model's test performance, suggesting
that our truthful valuation score can also be more robust to overfitting.

提出了一种新的数据评估方法，通过使用合适的评分规则和点间互信息测量来保证数据供应商总是通过真实报告观测到的数据来最大化他们的预期得分，该方法在贝叶斯机器学习环境中引入了一种新的点间互信息测量方法，证明了它在鲁棒性和对过拟合敏感性方面的优越性。