As machine learning systems become democratized, it becomes increasingly
important to help users easily debug their models. However, current data tools
are still primitive when it comes to helping users trace model performance
problems all the way to the data. We focus on the particular problem of slicing
data to identify subsets of the validation data where the model performs
poorly. This is an important problem in model validation because the overall
model performance can fail to reflect that of the smaller subsets, and slicing
allows users to analyze the model performance on a more granular-level. Unlike
general techniques (e.g., clustering) that can find arbitrary slices, our goal
is to find interpretable slices (which are easier to take action compared to
arbitrary subsets) that are problematic and large. We propose Slice Finder,
which is an interactive framework for identifying such slices using statistical
techniques. Applications include diagnosing model fairness and fraud detection,
where identifying slices that are interpretable to humans is crucial. This
research is part of a larger trend of Big data and Artificial Intelligence (AI)
integration and opens many opportunities for new research.

该研究提出了一个交互式框架 Slice Finder，使用统计技术找到可解释的子集去诊断模型的问题，应用包括识别模型公平性和欺诈检测等。

基于大数据和人工智能的模型验证自动数据切片方法

Automated Data Slicing for Model Validation:A Big data - AI Integration  Approach

Performance and high availability have become increasingly important drivers,
amongst other drivers, for user retention in the context of web services such
as social networks, and web search. Exogenic and/or endogenic factors often
give rise to anomalies, making it very challenging to maintain high
availability, while also delivering high performance. Given that
service-oriented architectures (SOA) typically have a large number of services,
with each service having a large set of metrics, automatic detection of
anomalies is non-trivial.
Although there exists a large body of prior research in anomaly detection,
existing techniques are not applicable in the context of social network data,
owing to the inherent seasonal and trend components in the time series data.
To this end, we developed two novel statistical techniques for automatically
detecting anomalies in cloud infrastructure data. Specifically, the techniques
employ statistical learning to detect anomalies in both application, and system
metrics. Seasonal decomposition is employed to filter the trend and seasonal
components of the time series, followed by the use of robust statistical
metrics -- median and median absolute deviation (MAD) -- to accurately detect
anomalies, even in the presence of seasonal spikes.
We demonstrate the efficacy of the proposed techniques from three different
perspectives, viz., capacity planning, user behavior, and supervised learning.
In particular, we used production data for evaluation, and we report Precision,
Recall, and F-measure in each case.

研究开发了两种新的统计技术，用于自动检测云基础架构数据中的异常，采用季节性分解和鲁棒统计度量，有效实现在季节性高峰存在的情况下的异常检测。从容量规划，用户行为和监督学习三个方面证明了所提出技术的有效性。