The rapid developments of various machine learning models and their
deployments in several applications has led to discussions around the
importance of looking beyond the accuracies of these models. Fairness of such
models is one such aspect that is deservedly gaining more attention. In this
work, we analyse the natural language representations of documents and
sentences (i.e., encodings) for any embedding-level bias that could potentially
also affect the fairness of the downstream tasks that rely on them. We identify
bias in these encodings either towards or against different sub-groups based on
the difference in their reconstruction errors along various subsets of
principal components. We explore and recommend ways to mitigate such bias in
the encodings while also maintaining a decent accuracy in classification models
that use them.

通过分析文档和句子的自然语言表示（即编码），我们发现并研究了这些编码中可能存在的针对不同子群体的偏见，并提出了缓解这种偏见的方法同时保持分类模型的较高准确性。

在公平表示和准确性之间填补差距

Closing the Gap in the Trade-off between Fair Representations and  Accuracy

Increasing model size when pretraining natural language representations often
results in improved performance on downstream tasks. However, at some point
further model increases become harder due to GPU/TPU memory limitations and
longer training times. To address these problems, we present two
parameter-reduction techniques to lower memory consumption and increase the
training speed of BERT. Comprehensive empirical evidence shows that our
proposed methods lead to models that scale much better compared to the original
BERT. We also use a self-supervised loss that focuses on modeling
inter-sentence coherence, and show it consistently helps downstream tasks with
multi-sentence inputs. As a result, our best model establishes new
state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having
fewer parameters compared to BERT-large. The code and the pretrained models are
available at this https URL

本文提出两种参数缩减技术，结合一个以自我监督方式处理句间连贯的损失函数，成功地使用更少的参数将 BERT 模型的性能扩展至其他基准测试集，包括 GLUE，RACE 和 SQuAD。

ALBERT: 自监督学习语言表示的轻量级 BERT

ALBERT: A Lite BERT for Self-supervised Learning of Language  Representations

Researchers have recently started investigating deep neural networks for
dialogue applications. In particular, generative sequence-to-sequence (Seq2Seq)
models have shown promising results for unstructured tasks, such as word-level
dialogue response generation. The hope is that such models will be able to
leverage massive amounts of data to learn meaningful natural language
representations and response generation strategies, while requiring a minimum
amount of domain knowledge and hand-crafting. An important challenge is to
develop models that can effectively incorporate dialogue context and generate
meaningful and diverse responses. In support of this goal, we review recently
proposed models based on generative encoder-decoder neural network
architectures, and show that these models have better ability to incorporate
long-term dialogue history, to model uncertainty and ambiguity in dialogue, and
to generate responses with high-level compositional structure.

该研究探讨了基于深度神经网络的自动生成响应的模型，着重讨论了结构松散的任务，如基于单词级别的对话响应生成，并提出了基于生成的编码器 - 解码器神经网络架构的新模型，以增强对话的长期历史纪录、建模对话中的不确定性和歧义性、生成具有高级组合结构的响应。