Imbalanced token distributions naturally exist in text documents, leading
neural language models to overfit on frequent tokens. The token imbalance may
dampen the robustness of radiology report generators, as complex medical terms
appear less frequently but reflect more medical information. In this study, we
demonstrate how current state-of-the-art models fail to generate infrequent
tokens on two standard benchmark datasets (IU X-RAY and MIMIC-CXR) of radiology
report generation. % However, no prior study has proposed methods to adapt
infrequent tokens for text generators feeding with medical images. To solve the
challenge, we propose the \textbf{T}oken \textbf{Im}balance Adapt\textbf{er}
(\textit{TIMER}), aiming to improve generation robustness on infrequent tokens.
The model automatically leverages token imbalance by an unlikelihood loss and
dynamically optimizes generation processes to augment infrequent tokens. We
compare our approach with multiple state-of-the-art methods on the two
benchmarks. Experiments demonstrate the effectiveness of our approach in
enhancing model robustness overall and infrequent tokens. Our ablation analysis
shows that our reinforcement learning method has a major effect in adapting
token imbalance for radiology report generation.

本研究提出了一种名为 TIMER 的方法来解决放射学报告生成中，由于词汇分布不均衡导致的常见词汇过拟合问题。该方法通过无似然损失自适应平衡并优化词汇生成过程，有效提高了模型的生成鲁棒性和生成不常见词的能力，同时与先进方法比较，使用强化学习得出的结果更为优越。

放射医学报告生成的令牌 imbalance 适应

Token Imbalance Adaptation for Radiology Report Generation

Token-level adaptive training approaches can alleviate the token imbalance
problem and thus improve neural machine translation, through re-weighting the
losses of different target tokens based on specific statistical metrics (e.g.,
token frequency or mutual information). Given that standard translation models
make predictions on the condition of previous target contexts, we argue that
the above statistical metrics ignore target context information and may assign
inappropriate weights to target tokens. While one possible solution is to
directly take target contexts into these statistical metrics, the
target-context-aware statistical computing is extremely expensive, and the
corresponding storage overhead is unrealistic. To solve the above issues, we
propose a target-context-aware metric, named conditional bilingual mutual
information (CBMI), which makes it feasible to supplement target context
information for statistical metrics. Particularly, our CBMI can be formalized
as the log quotient of the translation model probability and language model
probability by decomposing the conditional joint distribution. Thus CBMI can be
efficiently calculated during model training without any pre-specific
statistical calculations and large storage overhead. Furthermore, we propose an
effective adaptive training approach based on both the token- and
sentence-level CBMI. Experimental results on WMT14 English-German and WMT19
Chinese-English tasks show our approach can significantly outperform the
Transformer baseline and other related methods.

本研究提出一种基于条件双语互信息（CBMI）的目标上下文感知统计度量方法，用于改善神经机器翻译中令人困扰的令牌不平衡问题，有效提高模型性能，实验结果显示，我们的方法明显优于 Transformer 基线和其他相关方法。

基于条件双语互信息的自适应神经机器翻译训练

Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

There exists a token imbalance phenomenon in natural language as different
tokens appear with different frequencies, which leads to different learning
difficulties for tokens in Neural Machine Translation (NMT). The vanilla NMT
model usually adopts trivial equal-weighted objectives for target tokens with
different frequencies and tends to generate more high-frequency tokens and less
low-frequency tokens compared with the golden token distribution. However,
low-frequency tokens may carry critical semantic information that will affect
the translation quality once they are neglected. In this paper, we explored
target token-level adaptive objectives based on token frequencies to assign
appropriate weights for each target token during training. We aimed that those
meaningful but relatively low-frequency words could be assigned with larger
weights in objectives to encourage the model to pay more attention to these
tokens. Our method yields consistent improvements in translation quality on
ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain
more low-frequency tokens where we can get 1.68, 1.02, and 0.52 BLEU increases
compared with baseline, respectively. Further analyses show that our method can
also improve the lexical diversity of translation.

本文研究了神经机器翻译中的令牌不平衡现象，并提出采用基于目标令牌频率的目标令牌级自适应目标来训练模型，以提高翻译质量和提高翻译词汇的多样性。结果表明，与基线相比，在包含更多低频词汇的句子中，分别可以获得 1.68，1.02 和 0.52 的 BLEU 增益。