Most large language models (LLMs) are sensitive to prompts, and another
synonymous expression or a typo may lead to unexpected results for the model.
Composing an optimal prompt for a specific demand lacks theoretical support and
relies entirely on human experimentation, which poses a considerable obstacle
to popularizing generative artificial intelligence. However, there is no
systematic analysis of the stability of LLMs in resisting prompt perturbations
in real-world scenarios. In this work, we propose to evaluate the ease-of-use
of LLMs and construct E-Bench, simulating the actual situation of human use
from synonymous perturbation (including paraphrasing, simplification, and
colloquialism) and typographical perturbation (such as typing). On this basis,
we also discuss the combination of these two types of perturbation and analyze
the main reasons for performance degradation. Experimental results indicate
that with the increase of model size, although the ease-of-use are
significantly improved, there is still a long way to go to build a sufficiently
user-friendly model.

通过系统分析大型语言模型（LLMs）抵抗提示扰动的稳定性，本研究构建 E-Bench 模拟真实人类使用情况，并讨论了同义扰动和排版扰动的组合以及性能下降的主要原因，实验证明即便随着模型规模的增加，易用性显著提升，但仍需努力建立足够用户友好的模型。

E-Bench: 大型语言模型易用性评估

E-Bench: Towards Evaluating the Ease-of-Use of Large Language Models

We present the Keras domain packages KerasCV and KerasNLP, extensions of the
Keras API for Computer Vision and Natural Language Processing workflows,
capable of running on either JAX, TensorFlow, or PyTorch. These domain packages
are designed to enable fast experimentation, with a focus on ease-of-use and
performance. We adopt a modular, layered design: at the library's lowest level
of abstraction, we provide building blocks for creating models and data
preprocessing pipelines, and at the library's highest level of abstraction, we
provide pretrained ``task" models for popular architectures such as Stable
Diffusion, YOLOv8, GPT2, BERT, Mistral, CLIP, Gemma, T5, etc. Task models have
built-in preprocessing, pretrained weights, and can be fine-tuned on raw
inputs. To enable efficient training, we support XLA compilation for all
models, and run all preprocessing via a compiled graph of TensorFlow operations
using the tf.data API. The libraries are fully open-source (Apache 2.0 license)
and available on GitHub.

我们介绍了 KerasCV 和 KerasNLP 领域包，它们是 Keras API 的扩展，用于计算机视觉和自然语言处理工作流，在 JAX、TensorFlow 或 PyTorch 上运行。它们旨在实现快速实验，注重易用性和性能。