Big Data bring new opportunities to modern society and challenges to data
scientists. On one hand, Big Data hold great promises for discovering subtle
population patterns and heterogeneities that are not possible with small-scale
data. On the other hand, the massive sample size and high dimensionality of Big
Data introduce unique computational and statistical challenges, including
scalability and storage bottleneck, noise accumulation, spurious correlation,
incidental endogeneity, and measurement errors. These challenges are
distinguished and require new computational and statistical paradigm. This
article give overviews on the salient features of Big Data and how these
features impact on paradigm change on statistical and computational methods as
well as computing architectures. We also provide various new perspectives on
the Big Data analysis and computation. In particular, we emphasis on the
viability of the sparsest solution in high-confidence set and point out that
exogeneous assumptions in most statistical methods for Big Data can not be
validated due to incidental endogeneity. They can lead to wrong statistical
inferences and consequently wrong scientific conclusions.

这篇文献讨论了大数据的显著特征及其对统计和计算方法的范式变革产生的影响，强调了高置信度集中最稀疏解的可行性，并指出大多数针对大数据的统计方法中的外生性假设无法被验证，容易导致错误的统计推断和错误的科学结论。

大数据分析挑战

Challenges of Big Data Analysis

This paper deals with the computational complexity of conditions which
guarantee that the NP-hard problem of finding the sparsest solution to an
underdetermined linear system can be solved by efficient algorithms. In the
literature, several such conditions have been introduced. The most well-known
ones are the mutual coherence, the restricted isometry property (RIP), and the
nullspace property (NSP). While evaluating the mutual coherence of a given
matrix is easy, it has been suspected for some time that evaluating RIP and NSP
is computationally intractable in general. We confirm these conjectures by
showing that for a given matrix A and positive integer k, computing the best
constants for which the RIP or NSP hold is, in general, NP-hard. These results
are based on the fact that determining the spark of a matrix is NP-hard, which
is also established in this paper. Furthermore, we also give several complexity
statements about problems related to the above concepts.

本文主要研究了在怎样的条件下可以通过高效算法来解决 NP 难问题，这个问题是在欠定线性系统中发现最稀疏的解。通过研究出了互惠相干、限制等距性质和零空间特征来解决该问题，本文证实了计算 RIP 和 NSP 的最佳常数是 NP 难问题，并且讨论了与上述概念相关的几个复杂性陈述。