Tabular data -- also known as structured data -- is one of the most common
data forms in existence, thanks to the stable development and scaled deployment
of database systems in the last few decades. At present however, despite the
blast brought by large pre-trained models in other domains such as ChatGPT or
SAM, how can we extract common knowledge across tables at a scale that may
eventually lead to generalizable representation for tabular data remains a full
blank. Indeed, there have been a few works around this topic. Most (if not all)
of them are limited in the scope of a single table or fixed form of a schema.
In this work, we first identify the crucial research challenges behind tabular
data pre-training, particularly towards the cross-table scenario. We position
the contribution of this work in two folds: (i)-we collect and curate nearly 2k
high-quality tabular datasets, each of which is guaranteed to possess clear
semantics, clean labels, and other necessary meta information. (ii)-we propose
a novel framework that allows cross-table pre-training dubbed as CT-BERT.
Noticeably, in light of pioneering the scaled cross-table training, CT-BERT is
fully compatible with both supervised and self-supervised schemes, where the
specific instantiation of CT-BERT is very much dependent on the downstream
tasks. We further propose and implement a contrastive-learning-based and masked
table modeling (MTM) objective into CT-BERT, that is inspired from computer
vision and natural language processing communities but sophistically tailored
to tables. The extensive empirical results on 15 datasets demonstrate CT-BERT's
state-of-the-art performance, where both its supervised and self-supervised
setups significantly outperform the prior approaches.

该研究论文探讨了针对表格数据的交叉表预训练的关键研究挑战，并提出了一种名为 CT-BERT 的新型框架，同时引入了对比学习和表格建模的目标，通过广泛的实证结果证明 CT-BERT 在监督和自监督设置下显著优于之前的方法。

CT-BERT：通过跨表格预训练学习更好的表格表示

CT-BERT: Learning Better Tabular Representations Through Cross-Table  Pre-training

The COVID-19 pandemic has had a huge impact on various areas of human life.
Hence, the coronavirus pandemic and its consequences are being actively
discussed on social media. However, not all social media posts are truthful.
Many of them spread fake news that cause panic among readers, misinform people
and thus exacerbate the effect of the pandemic. In this paper, we present our
results at the Constraint@AAAI2021 Shared Task: COVID-19 Fake News Detection in
English. In particular, we propose our approach using the transformer-based
ensemble of COVID-Twitter-BERT (CT-BERT) models. We describe the models used,
the ways of text preprocessing and adding extra data. As a result, our best
model achieved the weighted F1-score of 98.69 on the test set (the first place
in the leaderboard) of this shared task that attracted 166 submitted teams in
total.

本文介绍了使用基于 Transformer 的 COVID-Twitter-BERT（CT-BERT）模型集合的方法来检测英文 COVID-19 虚假新闻的结果和表现。在约 166 个团队中，该方法在第一个共享任务的测试集上取得了最高的加权 F1 得分 98.69。