Balanced and efficient information flow is essential for optimizing language generation models. In this work, we propose Entropy-UID, a new token selection method that balances entropy and Uniform Information Density (UID) principles for enhanced efficiency of text generation. Our approach adaptively adjusts token selection by jointly minimizing entropy and surprisal, promoting more even information distribution across generated sequences. Theoretical validation demonstrates that Entropy-UID optimally reduces information spikes while maintaining fluency and coherence. The method has been evulated using information-theoretic metrics on multiple benchmark datasets, including WikiText-2, OpenWebText, and WMT. Experimental results show that Entropy-UID achieves lower surprisal and entropy variance compared to standard GPT-2 and alternative heuristics, leading to more balanced and human-like text generation. Our findings point towards the potential of leveraging information-theoretic constraints to refine token selection strategies in autoregressive language models.

本研究解决了语言生成模型中信息流不平衡和效率低下的问题。提出的熵-UID方法通过自适应调整标记选择，平衡熵和统一信息密度原则，从而实现更均衡的信息分布。实验结果显示，熵-UID在多个基准数据集上表现出更低的惊讶度和熵方差，推动了文本生成的自然性和流畅性。

熵-统一信息密度（Entropy-UID）：优化信息密度的方法