In this paper, we focus on generating a synthetic question answering (QA)
dataset using an adapted Translate-Align-Retrieve method. Using this method, we
created the largest Serbian QA dataset of more than 87K samples, which we name
SQuAD-sr. To acknowledge the script duality in Serbian, we generated both
Cyrillic and Latin versions of the dataset. We investigate the dataset quality
and use it to fine-tune several pre-trained QA models. Best results were
obtained by fine-tuning the BERTi\'c model on our Latin SQuAD-sr dataset,
achieving 73.91% Exact Match and 82.97% F1 score on the benchmark XQuAD
dataset, which we translated into Serbian for the purpose of evaluation. The
results show that our model exceeds zero-shot baselines, but fails to go beyond
human performance. We note the advantage of using a monolingual pre-trained
model over multilingual, as well as the performance increase gained by using
Latin over Cyrillic. By performing additional analysis, we show that questions
about numeric values or dates are more likely to be answered correctly than
other types of questions. Finally, we conclude that SQuAD-sr is of sufficient
quality for fine-tuning a Serbian QA model, in the absence of a manually
crafted and annotated dataset.

使用改进的翻译 - 对齐 - 检索方法生成了最大的塞尔维亚问答数据集 SQuAD-sr，并使用该数据集对几个预训练的问答模型进行了微调，最佳结果表明我们的模型超过了零 - shot 基线但未超越人类表现。

针对塞尔维亚语问答的 Transformer 模型的合成数据集创建和微调

Synthetic Dataset Creation and Fine-Tuning of Transformer Models for  Question Answering in Serbian

We present a new text-to-SQL dataset for electronic health records (EHRs).
The utterances were collected from 222 hospital staff, including physicians,
nurses, insurance review and health records teams, and more. To construct the
QA dataset on structured EHR data, we conducted a poll at a university hospital
and templatized the responses to create seed questions. Then, we manually
linked them to two open-source EHR databases, MIMIC-III and eICU, and included
them with various time expressions and held-out unanswerable questions in the
dataset, which were all collected from the poll. Our dataset poses a unique set
of challenges: the model needs to 1) generate SQL queries that reflect a wide
range of needs in the hospital, including simple retrieval and complex
operations such as calculating survival rate, 2) understand various time
expressions to answer time-sensitive questions in healthcare, and 3)
distinguish whether a given question is answerable or unanswerable based on the
prediction confidence. We believe our dataset, EHRSQL, could serve as a
practical benchmark to develop and assess QA models on structured EHR data and
take one step further towards bridging the gap between text-to-SQL research and
its real-life deployment in healthcare. EHRSQL is available at
this https URL.

我们提出了一种新的文本到 SQL 数据集，用于电子健康记录（EHRs）。该数据集提供了一个挑战，需要模型可以生成反映医院需求范围的 SQL 查询，包括简单检索和复杂操作，如计算生存率，同时需要理解时间表达式以回答与时间敏感相关的问题，并根据预测置信度区分问题是否有答案，以发展和评估结构化 EHR 数据上的 QA 模型并迈向文本到 SQL 研究和医疗保健方面部署之间的缩小差距。

电子病历文本到 SQL 的实用基准测试

EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

As large language models (LLMs) grow larger and more sophisticated, assessing
their "reasoning" capabilities in natural language grows more challenging.
Recent question answering (QA) benchmarks that attempt to assess reasoning are
often limited by a narrow scope of covered situations and subject matters. We
introduce WikiWhy, a QA dataset built around a novel auxiliary task: explaining
why an answer is true in natural language. WikiWhy contains over 9,000 "why"
question-answer-rationale triples, grounded on Wikipedia facts across a diverse
set of topics. Each rationale is a set of supporting statements connecting the
question to the answer. WikiWhy serves as a benchmark for the reasoning
capabilities of LLMs because it demands rigorous explicit rationales for each
answer to demonstrate the acquisition of implicit commonsense knowledge, which
is unlikely to be easily memorized. GPT-3 baselines achieve only 38.7%
human-evaluated correctness in the end-to-end answer & explain condition,
leaving significant room for future improvements.

通过构建一个包含九千多个问题 - 答案 - 理由三元组的 QA 数据集 “WikiWhy”，确定大型语言模型（LLMs）的推理能力，该数据集的每个理由都是一组支持语句，它们将问题和答案连接起来。 GPT-3 只能在结束的答案和解释条件下达到 38.7％的人类正确率，这为未来的改进留下了很大的空间。