Claude Shannon coined entropy to quantify the uncertainty of a random
distribution for communication coding theory. We observe that the uncertainty
nature of entropy also limits its direct usage in mathematical modeling.
Therefore we propose a new concept troenpy,as the canonical dual of entropy, to
quantify the certainty of the underlying distribution. We demonstrate two
applications in machine learning. The first is for the classical document
classification, we develop a troenpy based weighting scheme to leverage the
document class label. The second is a self-troenpy weighting scheme for
sequential data and show that it can be easily included in neural network based
language models and achieve dramatic perplexity reduction. We also define
quantum troenpy as the dual of the Von Neumann entropy to quantify the
certainty of quantum systems.

提出了新概念 troenpy，用于量化概率分布的确定性，以替代熵的应用，泛用于机器学习中，尤其在文本分类和序列数据处理方面表现突出。同时定义了量子 troenpy 以量化量子系统的确定性。

机器学习的确定性新信息理论

A New Information Theory of Certainty for Machine Learning

Shannon Information theory has achieved great success in not only
communication technology where it was originally developed for but also many
other science and engineering fields such as machine learning and artificial
intelligence. Inspired by the famous weighting scheme TF-IDF, we discovered
that information entropy has a natural dual. We complement the classical
Shannon information theory by proposing a novel quantity, namely troenpy.
Troenpy measures the certainty, commonness and similarity of the underlying
distribution. To demonstrate its usefulness, we propose a troenpy based
weighting scheme for document with class labels, namely positive class
frequency (PCF). On a collection of public datasets we show the PCF based
weighting scheme outperforms the classical TF-IDF and a popular Optimal
Transportation based word moving distance algorithm in a kNN setting. We
further developed a new odds-ratio type feature, namely Expected Class
Information Bias(ECIB), which can be regarded as the expected odds ratio of the
information quantity entropy and troenpy. In the experiments we observe that
including the new ECIB features and simple binary term features in a simple
logistic regression model can further significantly improve the performance.
The simple new weighting scheme and ECIB features are very effective and can be
computed with linear order complexity.

提出一种基于信息熵的新量度 troenpy 并给出其用于文本分类的权重方案 PCF；提出了期望类信息偏差 (ECIB) 特征，可以用于逻辑回归模型中，实验结果表明这些新方法能显著提高性能。