Information theoretic quantities play a central role in machine learning. The
recent surge in the complexity of data and models has increased the demand for
accurate estimation of these quantities. However, as the dimension grows the
estimation presents significant challenges, with existing methods struggling
already in relatively low dimensions. To address this issue, in this work, we
introduce $\texttt{REMEDI}$ for efficient and accurate estimation of
differential entropy, a fundamental information theoretic quantity. The
approach combines the minimization of the cross-entropy for simple, adaptive
base models and the estimation of their deviation, in terms of the relative
entropy, from the data density. Our approach demonstrates improvement across a
broad spectrum of estimation tasks, encompassing entropy estimation on both
synthetic and natural data. Further, we extend important theoretical
consistency results to a more generalized setting required by our approach. We
illustrate how the framework can be naturally extended to information theoretic
supervised learning models, with a specific focus on the Information Bottleneck
approach. It is demonstrated that the method delivers better accuracy compared
to the existing methods in Information Bottleneck. In addition, we explore a
natural connection between $\texttt{REMEDI}$ and generative modeling using
rejection sampling and Langevin dynamics.

通过简单、自适应的基模型的最小化和相对熵的估计，REMEDI 实现了高效且准确的差分熵估计方法，提升了信息论相关任务的准确性，并且在信息瓶颈方法中表现出更好的精度。