The purpose of this work is to describe the Orkg-Leaderboard software
designed to extract leaderboards defined as Task-Dataset-Metric tuples
automatically from large collections of empirical research papers in Artificial
Intelligence (AI). The software can support both the main workflows of
scholarly publishing, viz. as LaTeX files or as PDF files. Furthermore, the
system is integrated with the Open Research Knowledge Graph (ORKG) platform,
which fosters the machine-actionable publishing of scholarly findings. Thus the
system output, when integrated within the ORKG's supported Semantic Web
infrastructure of representing machine-actionable 'resources' on the Web,
enables: 1) broadly, the integration of empirical results of researchers across
the world, thus enabling transparency in empirical research with the potential
to also being complete contingent on the underlying data source(s) of
publications; and 2) specifically, enables researchers to track the progress in
AI with an overview of the state-of-the-art (SOTA) across the most common AI
tasks and their corresponding datasets via dynamic ORKG frontend views
leveraging tables and visualization charts over the machine-actionable data.
Our best model achieves performances above 90% F1 on the \textit{leaderboard}
extraction task, thus proving Orkg-Leaderboards a practically viable tool for
real-world usage. Going forward, in a sense, Orkg-Leaderboards transforms the
leaderboard extraction task to an automated digitalization task, which has
been, for a long time in the community, a crowdsourced endeavor.

描述了 Orkg-Leaderboard 软件的设计和应用，该软件能够从大量的人工智能实证研究论文中自动提取以任务 - 数据集 - 指标三元组定义的排行榜，并且能够支持学术出版的主要流程，以 LaTeX 文件或 PDF 文件的形式发布，同时结合 Open Research Knowledge Graph 平台实现与语义 Web 基础设施的有机整合，使得系统输出的信息能够被机器进行操作和理解。该应用还能追踪 AI 发展的进程和状态，提供实时动态的信息服务。

ORKG-Leaderboards：基于知识图谱的排行榜挖掘系统工作流程

ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a  Knowledge Graph

We present a large-scale empirical investigation of the zero-shot learning
phenomena in a specific recognizing textual entailment (RTE) task category,
i.e. the automated mining of leaderboards for Empirical AI Research. The prior
reported state-of-the-art models for leaderboards extraction formulated as an
RTE task, in a non-zero-shot setting, are promising with above 90% reported
performances. However, a central research question remains unexamined: did the
models actually learn entailment? Thus, for the experiments in this paper, two
prior reported state-of-the-art models are tested out-of-the-box for their
ability to generalize or their capacity for entailment, given leaderboard
labels that were unseen during training. We hypothesize that if the models
learned entailment, their zero-shot performances can be expected to be
moderately high as well--perhaps, concretely, better than chance. As a result
of this work, a zero-shot labeled dataset is created via distant labeling
formulating the leaderboard extraction RTE task.

这篇论文探究了零样本学习的现象在特定的文本蕴涵识别 (RTE) 任务类别中的应用，即自动挖掘带有实验证据的人工智能研究排行榜。作者通过实验检验两个现有最先进的模型是否具有泛化能力或者蕴涵的显现能力，进一步提出了零样本标签数据集的构建方法。