If a question cannot be answered with the available information, robust
systems for question answering (QA) should know _not_ to answer. One way to
build QA models that do this is with additional training data comprised of
unanswerable questions, created either by employing annotators or through
automated methods for unanswerable question generation. To show that the model
complexity of existing automated approaches is not justified, we examine a
simpler data augmentation method for unanswerable question generation in
English: performing antonym and entity swaps on answerable questions. Compared
to the prior state-of-the-art, data generated with our training-free and
lightweight strategy results in better models (+1.6 F1 points on SQuAD 2.0 data
with BERT-large), and has higher human-judged relatedness and readability. We
quantify the raw benefits of our approach compared to no augmentation across
multiple encoder models, using different amounts of generated data, and also on
TydiQA-MinSpan data (+9.3 F1 points with BERT-large). Our results establish
swaps as a simple but strong baseline for future work.

一种用于问题回答的数据增强方法，通过对可回答问题进行反义词和实体交换生成不可回答的问题，相比于现有的自动化方法，该方法具有更好的模型性能和更高的人类判断相关性和可读性，并被验证为未来工作的简单但强大的基准。

一个生成无法回答问题的轻量级方法

A Lightweight Method to Generate Unanswerable Questions in English

In this work, we study the large-scale pretraining of BERT-Large with
differentially private SGD (DP-SGD). We show that combined with a careful
implementation, scaling up the batch size to millions (i.e., mega-batches)
improves the utility of the DP-SGD step for BERT; we also enhance its
efficiency by using an increasing batch size schedule. Our implementation
builds on the recent work of [SVK20], who demonstrated that the overhead of a
DP-SGD step is minimized with effective use of JAX [BFH+18, FJL18] primitives
in conjunction with the XLA compiler [XLA17]. Our implementation achieves a
masked language model accuracy of 60.5% at a batch size of 2M, for $\epsilon =
5.36$. To put this number in perspective, non-private BERT models achieve an
accuracy of $\sim$70%.

本文探究了使用差分隐私 SGD（DP-SGD）进行 BERT-Large 的大规模预训练。通过仔细的实现和批量大小增加，我们提高了 DP-SGD 步骤的实用性，并使用增加的批量大小计划提高了其效率。

大规模差分隐私 BERT

Large-Scale Differentially Private BERT

The quality of non-default ranking on e-commerce platforms, such as based on
ascending item price or descending historical sales volume, often suffers from
acute relevance problems, since the irrelevant items are much easier to be
exposed at the top of the ranking results. In this work, we propose a two-stage
ranking scheme, which first recalls wide range of candidate items through
refined query/title keyword matching, and then classifies the recalled items
using BERT-Large fine-tuned on human label data. We also implemented parallel
prediction on multiple GPU hosts and a C++ tokenization custom op of
Tensorflow. In this data challenge, our model won the 1st place in the
supervised phase (based on overall F1 score) and 2nd place in the final phase
(based on average per query F1 score).

本文提出了一种基于 Bert-Large 的双阶段排序方案，首先通过精细的查询 / 标题关键词匹配检索候选商品，然后利用经过人工标注数据的 BERT-Large 进行分类，同时在多个 GPU 主机上实现并行预测和基于 Tensorflow 的 C++ 分词自定义操作。在数据挑战中，我们的模型在监督阶段的总 F1 分数上获得了第一名，在最终阶段的平均每个查询的 F1 分数上获得了第二名。

针对电子商务非默认搜索排名的 BERT 微调

Fine-tune BERT for E-commerce Non-Default Search Ranking

When answering a question, people often draw upon their rich world knowledge
in addition to the particular context. Recent work has focused primarily on
answering questions given some relevant document or context, and required very
little general background. To investigate question answering with prior
knowledge, we present CommonsenseQA: a challenging new dataset for commonsense
question answering. To capture common sense beyond associations, we extract
from ConceptNet (Speer et al., 2017) multiple target concepts that have the
same semantic relation to a single source concept. Crowd-workers are asked to
author multiple-choice questions that mention the source concept and
discriminate in turn between each of the target concepts. This encourages
workers to create questions with complex semantics that often require prior
knowledge. We create 12,247 questions through this procedure and demonstrate
the difficulty of our task with a large number of strong baselines. Our best
baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy,
well below human performance, which is 89%.

该论文提出了一个常识问答的新数据集 ——CommonsenseQA，并采用了多个目标概念，旨在提高常识推理的难度，在使用 BERT-large 作为基线方法的情况下，最佳准确率为 56%。