The instruction learning paradigm -- where a model learns to perform new
tasks from task descriptions alone -- has become popular in general-purpose
model research. The capabilities of large transformer models as instruction
learners, however, remain poorly understood. We use a controlled synthetic
environment to characterize such capabilities. Specifically, we use the task of
deciding whether a given string matches a regular expression (viewed as an
instruction) to identify properties of tasks, instructions, and instances that
make instruction learning challenging. For instance, we find that our model, a
fine-tuned T5-based text2text transformer, struggles with large regular
languages, suggesting that less precise instructions are challenging for
models. Additionally, instruction executions that require tracking longer
contexts of prior steps are also more difficult. We use our findings to
systematically construct a challenging instruction learning dataset, which we
call Hard RegSet. Fine-tuning on Hard RegSet, our large transformer learns to
correctly interpret only 65.6% of test instructions (with at least 90%
accuracy), and 11%-24% of the instructions in out-of-distribution
generalization settings. We propose Hard RegSet as a challenging instruction
learning task, and a controlled environment for studying instruction learning.

使用 T5-based text2text transformer 构建了一个 Hard RegSet dataset，用于研究 instruction learning，该 transformer 学习从指示中执行任务的能力存在挑战，其中大型正则表达式和需要跟踪更长的先前步骤的执行更加困难。

什么使得指导学习变得困难？一个人工环境的研究和新挑战

What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment

In human-human conversations, Context Tracking deals with identifying
important entities and keeping track of their properties and relationships.
This is a challenging problem that encompasses several subtasks such as slot
tagging, coreference resolution, resolving plural mentions and entity linking.
We approach this problem as an end-to-end modeling task where the
conversational context is represented by an entity repository containing the
entity references mentioned so far, their properties and the relationships
between them. The repository is updated turn-by-turn, thus making training and
inference computationally efficient even for long conversations. This paper
lays the groundwork for an investigation of this framework in two ways. First,
we release Contrack, a large scale human-human conversation corpus for context
tracking with people and location annotations. It contains over 7000
conversations with an average of 11.8 turns, 5.8 entities and 15.2 references
per conversation. Second, we open-source a neural network architecture for
context tracking. Finally we compare this network to state-of-the-art
approaches for the subtasks it subsumes and report results on the involved
tradeoffs.

本文提出了一种实时更新的实体仓库，用于表示对话上下文中已经提到的实体及其关系，并采用端到端建模方法解决了包括标签标注、指代消解、实体链接等多个子任务，同时提供了一个大规模对话语料 Contrack ，以及一种基于神经网络的上下文跟踪方法，并与现有方法进行比较。