Shannon entropy is often a quantity of interest to linguists studying the
communicative capacity of human language. However, entropy must typically be
estimated from observed data because researchers do not have access to the
underlying probability distribution that gives rise to these data. While
entropy estimation is a well-studied problem in other fields, there is not yet
a comprehensive exploration of the efficacy of entropy estimators for use with
linguistic data. In this work, we fill this void, studying the empirical
effectiveness of different entropy estimators for linguistic distributions. In
a replication of two recent information-theoretic linguistic studies, we find
evidence that the reported effect size is over-estimated due to over-reliance
on poor entropy estimators. Finally, we end our paper with concrete
recommendations for entropy estimation depending on distribution type and data
availability.

本文研究了 Shannon 熵在语言学任务中的应用，重点关注从可观察的语言数据中估算熵的效果，并通过两个信息论语言学研究的复制实验发现，由于过度依赖不良熵估算器，报告的效应大小被高估了。最后我们提出了针对不同类型分布和数据可用性的熵估算建议。

估计语言分布的熵

Estimating the Entropy of Linguistic Distributions

We propose a test of independence of two multivariate random vectors, given a
sample from the underlying population. Our approach, which we call MINT, is
based on the estimation of mutual information, whose decomposition into joint
and marginal entropies facilitates the use of recently-developed efficient
entropy estimators derived from nearest neighbour distances. The proposed
critical values, which may be obtained from simulation (in the case where one
marginal is known) or resampling, guarantee that the test has nominal size, and
we provide local power analyses, uniformly over classes of densities whose
mutual information satisfies a lower bound. Our ideas may be extended to
provide a new goodness-of-fit tests of normal linear models based on assessing
the independence of our vector of covariates and an appropriately-defined
notion of an error vector. The theory is supported by numerical studies on both
simulated and real data.

提出了一种称为 MINT 的新方法，该方法基于互信息的估计，并使用从最近的邻居距离导出的有效熵估计器，从而便于将互信息分解为联合和边际熵，用于检验两个多变量随机向量的独立性，并可将其扩展为基于测量协变量向量与误差向量独立性的新正态线性模型是否适配的卡方检验，并在模拟和真实数据上进行了数值研究。