Calculating the capacity (with or without feedback) of channels with memory
and continuous alphabets is a challenging task. It requires optimizing the
directed information (DI) rate over all channel input distributions. The
objective is a multi-letter expression, whose analytic solution is only known
for a few specific cases. When no analytic solution is present or the channel
model is unknown, there is no unified framework for calculating or even
approximating capacity. This work proposes a novel capacity estimation
algorithm that treats the channel as a `black-box', both when feedback is or is
not present. The algorithm has two main ingredients: (i) a neural distribution
transformer (NDT) model that shapes a noise variable into the channel input
distribution, which we are able to sample, and (ii) the DI neural estimator
(DINE) that estimates the communication rate of the current NDT model. These
models are trained by an alternating maximization procedure to both estimate
the channel capacity and obtain an NDT for the optimal input distribution. The
method is demonstrated on the moving average additive Gaussian noise channel,
where it is shown that both the capacity and feedback capacity are estimated
without knowledge of the channel transition kernel. The proposed estimation
framework opens the door to a myriad of capacity approximation results for
continuous alphabet channels that were inaccessible until now.

利用神经分布转换器（NDT）模型对连续字母信道进行容量估计，从而推导出一种新的估计框架来逐步最大化目标函数中的分数，以便进行准确的估计。

通过有向信息神经估计器计算具有记忆的连续信道容量

Capacity of Continuous Channels with Memory via Directed Information  Neural Estimator

The problem of estimating the directed information rate between two discrete
processes $\{X_n\}$ and $\{Y_n\}$ via the plug-in (or maximum-likelihood)
estimator is considered. When the joint process $\{(X_n,Y_n)\}$ is a Markov
chain of a given memory length, the plug-in estimator is shown to be
asymptotically Gaussian and to converge at the optimal rate $O(1/\sqrt{n})$
under appropriate conditions; this is the first estimator that has been shown
to achieve this rate. An important connection is drawn between the problem of
estimating the directed information rate and that of performing a hypothesis
test for the presence of causal influence between the two processes. Under
fairly general conditions, the null hypothesis, which corresponds to the
absence of causal influence, is equivalent to the requirement that the directed
information rate be equal to zero. In that case a finer result is established,
showing that the plug-in converges at the faster rate $O(1/n)$ and that it is
asymptotically $\chi^2$-distributed. This is proved by showing that this
estimator is equal to (a scalar multiple of) the classical likelihood ratio
statistic for the above hypothesis test. Finally it is noted that these results
facilitate the design of an actual likelihood ratio test for the presence or
absence of causal influence.

研究了在联合过程是给定记忆长度的马尔可夫链的情况下，通过插件（或最大似然）估计算法来估算两个离散过程 {Xn} 和 {Yn} 之间的有向信息速率的问题。插件估计器被证明是渐近高斯的，并在适当条件下以最优速率 O（1/√n）收敛；这是第一个已被证明可以达到此速率的估计器。同时，该研究发现估算有向信息速率的问题与执行关于两个过程之间因果关系存在的假设检验问题之间存在重要联系。 这些结果有助于设计实际的似然比检验，以检验因果影响的存在或不存在。

估计被指导的信息和检验因果关系

Estimating the Directed Information and Testing for Causality

Four estimators of the directed information rate between a pair of jointly
stationary ergodic finite-alphabet processes are proposed, based on universal
probability assignments. The first one is a Shannon--McMillan--Breiman type
estimator, similar to those used by Verd\'u (2005) and Cai, Kulkarni, and
Verd\'u (2006) for estimation of other information measures. We show the almost
sure and $L_1$ convergence properties of the estimator for any underlying
universal probability assignment. The other three estimators map universal
probability assignments to different functionals, each exhibiting relative
merits such as smoothness, nonnegativity, and boundedness. We establish the
consistency of these estimators in almost sure and $L_1$ senses, and derive
near-optimal rates of convergence in the minimax sense under mild conditions.
These estimators carry over directly to estimating other information measures
of stationary ergodic finite-alphabet processes, such as entropy rate and
mutual information rate, with near-optimal performance and provide alternatives
to classical approaches in the existing literature. Guided by these theoretical
results, the proposed estimators are implemented using the context-tree
weighting algorithm as the universal probability assignment. Experiments on
synthetic and real data are presented, demonstrating the potential of the
proposed schemes in practice and the utility of directed information estimation
in detecting and measuring causal influence and delay.

提出了四种估计器用于有限字母过程对之间的有向信息速率，基于通用概率分配。利用这些理论结果，使用上下文树加权算法作为通用概率分配实现了提出的估计器，并在合成和实际数据上进行了实验，证明了提出的方案的潜力及有向信息估计在检测和测量因果影响和延迟方面的实用性。