The widespread use of spreadsheet environments by billions of users presents
a unique opportunity for formula-authoring assistance. Although large language
models, such as Codex, can assist in general-purpose languages, they are
expensive to train and challenging to deploy due to their large model sizes (up
to billions of parameters). Moreover, they require hundreds of gigabytes of
training data. We present FLAME, a T5-based model trained on Excel formulas
that leverages domain insights to achieve competitive performance with a
substantially smaller model (60M parameters) and two orders of magnitude less
training data. We curate a training dataset using sketch deduplication,
introduce an Excel-specific formula tokenizer for our model, and use
domain-specific versions of masked span prediction and noisy auto-encoding as
pretraining objectives. We evaluate FLAME on formula repair, formula
auto-completion, and a novel task called syntax reconstruction. FLAME (60M) can
outperform much larger models, such as Codex-Davinci (175B), Codex-Cushman
(12B), and CodeT5 (220M), in 6 out of 10 settings.

FLAME 是一个基于 T5 的模型，通过在 Excel 公式中利用领域洞察力， using sketch deduplication，introduce an Excel-specific formula tokenizer and use domain-specific versions of masked span prediction and noisy auto-encoding aspretraining objectives，将模型数量大幅减少（60M parameters）和训练数据量两个数量级。FLAME 优于 Codex-Davinci（175B），Codex-Cushman（12B）和 CodeT5（220M）等较大的模型，可广泛应用于公式修复、公式自动完成以及语法重构等方面。

FLAME：适用于电子表格公式的小型语言模型

FLAME: A small language model for spreadsheet formulas

Spreadsheet formula prediction has been an important program synthesis
problem with many real-world applications. Previous works typically utilize
input-output examples as the specification for spreadsheet formula synthesis,
where each input-output pair simulates a separate row in the spreadsheet.
However, this formulation does not fully capture the rich context in real-world
spreadsheets. First, spreadsheet data entries are organized as tables, thus
rows and columns are not necessarily independent from each other. In addition,
many spreadsheet tables include headers, which provide high-level descriptions
of the cell data. However, previous synthesis approaches do not consider
headers as part of the specification. In this work, we present the first
approach for synthesizing spreadsheet formulas from tabular context, which
includes both headers and semi-structured tabular data. In particular, we
propose SpreadsheetCoder, a BERT-based model architecture to represent the
tabular context in both row-based and column-based formats. We train our model
on a large dataset of spreadsheets, and demonstrate that SpreadsheetCoder
achieves top-1 prediction accuracy of 42.51%, which is a considerable
improvement over baselines that do not employ rich tabular context. Compared to
the rule-based system, SpreadsheetCoder assists 82% more users in composing
formulas on Google Sheets.

通过使用半结构化表格数据和表头，我们提出了第一个从表格语境中综合电子表格公式的方法，包括行和列格式的电子表格中的表格上下文。具体而言，我们提出了基于 BERT 的模型体系结构 SpreadsheetCoder，在大型电子表格数据集上对我们的模型进行了训练，并证明了其在表格上下文中表现出的高准确度。