Transformer based language models (LMs) demonstrate increasing performance
with scale across a wide variety of tasks. Scale alone however cannot enable
models to solve tasks that require access to ephemeral, changing, or private
data that was unavailable at training time. Many useful tasks may also benefit
from LMs being able to access APIs that read or modify state. In this work, we
present Tool Augmented Language Models (TALM), combining a text-only approach
to augment language models with non-differentiable tools, and an iterative
"self-play" technique to bootstrap performance starting from few tool
demonstrations. TALM exhibits strong performance on both a knowledge-heavy QA
task and a reasoning oriented math task with simple tools. At a given model
scale, TALM significantly outperforms non-augmented LMs. We further demonstrate
that TALM successfully performs out-of-distribution inferences on both QA and
math tasks, where non-augmented LMs fail. Our results suggest that Tool
Augmented Language Models are a promising direction to enrich LMs'
capabilities, with less dependence on scale.

本文介绍了一种基于迭代 “自我对弈” 技术的文本增强语言模型方法，使用不可微分的工具扩充语言模型功能，成功在知识丰富型问答和简单工具所需的数学任务中具有很强的表现力，优于非增强型语言模型，在 QA 和数学任务的超越分布推理方面更是取得了成功，证明了工具增强型语言模型是一种非常有前景的方法，可以使语言模型在不依赖于模型（尺度）的基础上具备更多的能力。

TALM: 工具增强型语言模型

TALM: Tool Augmented Language Models

As the issues of privacy and trust are receiving increasing attention within
the research community, various attempts have been made to anonymize textual
data. A significant subset of these approaches incorporate differentially
private mechanisms to perturb word embeddings, thus replacing individual words
in a sentence. While these methods represent very important contributions, have
various advantages over other techniques and do show anonymization
capabilities, they have several shortcomings. In this paper, we investigate
these weaknesses and demonstrate significant mathematical constraints
diminishing the theoretical privacy guarantee as well as major practical
shortcomings with regard to the protection against deanonymization attacks, the
preservation of content of the original sentences as well as the quality of the
language output. Finally, we propose a new method for text anonymization based
on transformer based language models fine-tuned for paraphrasing that
circumvents most of the identified weaknesses and also offers a formal privacy
guarantee. We evaluate the performance of our method via thorough
experimentation and demonstrate superior performance over the discussed
mechanisms.

本文介绍了一个新的文本匿名处理方法，该方法使用 fine-tuned 变压器语言模型实现了高质量的语言输出，同时避免了匿名化方法存在的隐私保护及文本保留方面的问题。