Many computational linguistic methods have been proposed to study the
information content of languages. We consider two interesting research
questions: 1) how is information distributed over long documents, and 2) how
does content reduction, such as token selection and text summarization, affect
the information density in long documents. We present four criteria for
information density estimation for long documents, including surprisal,
entropy, uniform information density, and lexical density. Among those
criteria, the first three adopt the measures from information theory. We
propose an attention-based word selection method for clinical notes and study
machine summarization for multiple-domain documents. Our findings reveal the
systematic difference in information density of long text in various domains.
Empirical results on automated medical coding from long clinical notes show the
effectiveness of the attention-based word selection method.

我们提出了四项用于估计长文档信息密度的标准，包括惊讶度、熵、均匀信息密度和词汇密度，并提出了一种基于注意力的临床笔记词语选择方法和多领域文档的机器摘要方法。实证结果表明不同领域的长文本信息密度存在系统差异，基于注意力的词语选择方法对于长临床笔记的自动医学编码具有有效性。