Current natural language understanding (NLU) models have been continuously
scaling up, both in terms of model size and input context, introducing more
hidden and input neurons. While this generally improves performance on average,
the extra neurons do not yield a consistent improvement for all instances. This
is because some hidden neurons are redundant, and the noise mixed in input
neurons tends to distract the model. Previous work mainly focuses on
extrinsically reducing low-utility neurons by additional post- or
pre-processing, such as network pruning and context selection, to avoid this
problem. Beyond that, can we make the model reduce redundant parameters and
suppress input noise by intrinsically enhancing the utility of each neuron? If
a model can efficiently utilize neurons, no matter which neurons are ablated
(disabled), the ablated submodel should perform no better than the original
full model. Based on such a comparison principle between models, we propose a
cross-model comparative loss for a broad range of tasks. Comparative loss is
essentially a ranking loss on top of the task-specific losses of the full and
ablated models, with the expectation that the task-specific loss of the full
model is minimal. We demonstrate the universal effectiveness of comparative
loss through extensive experiments on 14 datasets from 3 distinct NLU tasks
based on 4 widely used pretrained language models, and find it particularly
superior for models with few parameters or long input.

本研究提出了一种基于跨模型比较损失的方法，通过提高神经元的效率和减少输入的噪声来优化自然语言理解（NLU）模型，对 14 个数据集进行了广泛实验，证明跨模型比较损失对于因具有少量参数或长输入而难以优化的模型尤为有效。

跨模型比较损失函数用于提升语言理解神经元效用

Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding

The overreliance on large parallel corpora significantly limits the
applicability of machine translation systems to the majority of language pairs.
Back-translation has been dominantly used in previous approaches for
unsupervised neural machine translation, where pseudo sentence pairs are
generated to train the models with a reconstruction loss. However, the pseudo
sentences are usually of low quality as translation errors accumulate during
training. To avoid this fundamental issue, we propose an alternative but more
effective approach, extract-edit, to extract and then edit real sentences from
the target monolingual corpora. Furthermore, we introduce a comparative
translation loss to evaluate the translated target sentences and thus train the
unsupervised translation systems. Experiments show that the proposed approach
consistently outperforms the previous state-of-the-art unsupervised machine
translation systems across two benchmarks (English-French and English-German)
and two low-resource language pairs (English-Romanian and English-Russian) by
more than 2 (up to 3.63) BLEU points.

提出并实现了一种基于抽取 - 编辑方法的无监督神经机器翻译，与以往仅使用错误积累法的基准性方法相比，在多个语言对和领域（包括低资源语言）的实验中表现更加优异。