This work presents BanglaNLG, a comprehensive benchmark for evaluating
natural language generation (NLG) models in Bangla, a widely spoken yet
low-resource language. We aggregate six challenging conditional text generation
tasks under the BanglaNLG benchmark, introducing a new dataset on dialogue
generation in the process. Furthermore, using a clean corpus of 27.5 GB of
Bangla data, we pretrain BanglaT5, a sequence-to-sequence Transformer language
model for Bangla. BanglaT5 achieves state-of-the-art performance in all of
these tasks, outperforming several multilingual models by up to 9% absolute
gain and 32% relative gain. We are making the new dialogue dataset and the
BanglaT5 model publicly available at this https URL in
the hope of advancing future research on Bangla NLG.

本文提出 BanglaNLG，用于评估 Bangla 自然语言生成（NLG）模型的全面基准，并介绍了六个具有挑战性的条件文本生成任务和一个新的对话生成数据集。利用 27.5 GB 干净的 Bangla 数据集，预训练了 BanglaT5，一种面向 Bangla 的序列到序列 Transformer 语言模型。 BanglaT5 在所有任务中均达到最先进的性能，比多语言模型高出 9％的绝对收益和 32％的相对收益。我们将新的对话数据集和 BanglaT5 模型公开发布，以期推动未来的 Bangla NLG 研究。

BanglaNLG 和 BanglaT5：用于评估孟加拉低资源自然语言生成的基准和资源

BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla

We study the feasibility of a Data Science assistant powered by a
sequence-to-sequence transformer by training a new model JuPyT5 on all publicly
available Jupyter Notebook GitHub repositories and developing a new metric:
Data Science Problems (DSP). DSP is a collection of 1119 problems curated from
306 pedagogical notebooks with 92 dataset dependencies, natural language and
Markdown problem descriptions, and assert-based unit tests. These notebooks
were designed to test university students' mastery of various Python
implementations of Math and Data Science, and we now leverage them to study the
ability of JuPyT5 to understand and pass the tests. We analyze the content of
DSP, validate its quality, and we find that given 100 sampling attempts JuPyT5
is able to solve 77.5\% of the DSP problems. We further present various
ablation and statistical analyses and compare DSP to other recent natural
language to code benchmarks.

本论文研究了一个由序列到序列变换器驱动的数据科学助手的可行性，通过使用所有公开可用的 Jupyter Notebook GitHub 存储库对一个名为 JuPyT5 的新模型进行训练。新模型根据 306 个教学笔记本上的 1119 个 Python 实现的数学和数据科学问题进行评估。实验发现 JuPyT5 可以解决 77.5％的数据科学问题，同时对 DSP 进行了比较分析。