We empirically demonstrate that a transformer pre-trained on country-scale
unlabeled human mobility data learns embeddings capable, through fine-tuning,
of developing a deep understanding of the target geography and its
corresponding mobility patterns. Utilizing an adaptation framework, we evaluate
the performance of our pre-trained embeddings in encapsulating a broad spectrum
of concepts directly and indirectly related to human mobility. This includes
basic notions, such as geographic location and distance, and extends to more
complex constructs, such as administrative divisions and land cover. Our
extensive empirical analysis reveals a substantial performance boost gained
from pre-training, reaching up to 38% in tasks such as tree-cover regression.
We attribute this result to the ability of the pre-training to uncover
meaningful patterns hidden in the raw data, beneficial for modeling relevant
high-level concepts. The pre-trained embeddings emerge as robust
representations of regions and trajectories, potentially valuable for a wide
range of downstream applications.

基于国家级无标签人类流动数据进行事先训练的转换器能够通过微调形成对目标地理区域及其相应流动模式的深入理解，我们的预先训练嵌入在涵盖与人类流动直接和间接相关的广泛概念方面表现出良好性能，包括地理位置、距离、行政区划和土地覆盖等，通过大量实证分析，我们发现预先训练可以显著提升性能，在树木覆盖回归等任务中达到 38％，我们将这一结果归因于预先训练能够揭示原始数据中隐藏的有意义模式，并有助于建模相关的高级概念，预先训练的嵌入成为表征区域和轨迹的强大工具，在广泛的后续应用中具有潜在价值。

预训练的 Transformer 揭示人类移动数据中有意义的模式

Pre-trained Transformer Uncovers Meaningful Patterns in Human Mobility  Data

An important task for designing QA systems is answer sentence selection
(AS2): selecting the sentence containing (or constituting) the answer to a
question from a set of retrieved relevant documents. In this paper, we propose
three novel sentence-level transformer pre-training objectives that incorporate
paragraph-level semantics within and across documents, to improve the
performance of transformers for AS2, and mitigate the requirement of large
labeled datasets. Specifically, the model is tasked to predict whether: (i) two
sentences are extracted from the same paragraph, (ii) a given sentence is
extracted from a given paragraph, and (iii) two paragraphs are extracted from
the same document. Our experiments on three public and one industrial AS2
datasets demonstrate the empirical superiority of our pre-trained transformers
over baseline models such as RoBERTa and ELECTRA for AS2.

本文提出了三种新的句子级别的 transformer 预训练目标，以提高 transformer 模型在答案句子选择和文档检索任务中的表现，并减轻对大量标记数据的需求。实验证明，我们的预训练 transformer 模型在三个公共 AS2 数据集和一个工业 AS2 数据集上的表现优越，胜过 RoBERTa 和 ELECTRA 等基线模型。