Recent advances in large language models have prompted researchers to examine
their abilities across a variety of linguistic tasks, but little has been done
to investigate how models handle the interactions in meaning across words and
larger syntactic forms -- i.e. phenomena at the intersection of syntax and
semantics. We present the semantic notion of agentivity as a case study for
probing such interactions. We created a novel evaluation dataset by utilitizing
the unique linguistic properties of a subset of optionally transitive English
verbs. This dataset was used to prompt varying sizes of three model classes to
see if they are sensitive to agentivity at the lexical level, and if they can
appropriately employ these word-level priors given a specific syntactic
context. Overall, GPT-3 text-davinci-003 performs extremely well across all
experiments, outperforming all other models tested by far. In fact, the results
are even better correlated with human judgements than both syntactic and
semantic corpus statistics. This suggests that LMs may potentially serve as
more useful tools for linguistic annotation, theory testing, and discovery than
select corpora for certain tasks.

以 semantic notion of agentivity 为例，研究了 large language models 在句法和语义交互方面的表现；作者创建了一个新颖的评估数据集，发现 GPT-3 的表现非常好，甚至比语法和语义语料库的统计结果还要好，这表明大语言模型可能对语言注释、理论测试和发现比某些任务的语料库更有用。

句法与语义的交汇：通过代理性探究语言模型的句法 - 语义接口

Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics  Interface of LMs Through Agentivity

We describe two systems that use text-davinci-003, a large language model,
for the automatized correction of (i) exercises in translating back and forth
between natural language and the languages of propositional logic and
first-order predicate logic and (ii) exercises in writing simple arguments in
natural language in non-mathematical scenarios.

使用文本 - Davinci-003 大型语言模型，自动纠正转换自然语言和命题逻辑、一阶谓词逻辑语言间翻译练习，以及在非数学场景下写作简单论证的练习。

使用大型语言模型进行形式化 / 非形式化及自然论证练习，面向初学者

Using large language models for (de-)formalization and natural  argumentation exercises for beginner's students

The global economy is increasingly dependent on knowledge workers to meet the
needs of public and private organizations. While there is no single definition
of knowledge work, organizations and industry groups still attempt to measure
individuals' capability to engage in it. The most comprehensive assessment of
capability readiness for professional knowledge workers is the Uniform CPA
Examination developed by the American Institute of Certified Public Accountants
(AICPA). In this paper, we experimentally evaluate OpenAI's `text-davinci-003`
and prior versions of GPT on both a sample Regulation (REG) exam and an
assessment of over 200 multiple-choice questions based on the AICPA Blueprints
for legal, financial, accounting, technology, and ethical tasks. First, we find
that `text-davinci-003` achieves a correct rate of 14.4% on a sample REG exam
section, significantly underperforming human capabilities on quantitative
reasoning in zero-shot prompts. Second, `text-davinci-003` appears to be
approaching human-level performance on the Remembering & Understanding and
Application skill levels in the Exam absent calculation. For best prompt and
parameters, the model answers 57.6% of questions correctly, significantly
better than the 25% guessing rate, and its top two answers are correct 82.1% of
the time, indicating strong non-entailment. Finally, we find that recent
generations of GPT-3 demonstrate material improvements on this assessment,
rising from 30% for `text-davinci-001` to 57% for `text-davinci-003`. These
findings strongly suggest that large language models have the potential to
transform the quality and efficiency of future knowledge work.

本文研究了 text-davinci-003 和不同版本的 GPT-3 的表现，发现大语言模型有潜力提升未来知识工作的质量和效率。