The limits of open-ended generative models are unclear, yet increasingly
important. What causes them to succeed and what causes them to fail? In this
paper, we take a prompt-centric approach to analyzing and bounding the
abilities of open-ended generative models. We present a generic methodology of
analysis with two challenging prompt constraint types: structural and
stylistic. These constraint types are categorized into a set of well-defined
constraints that are analyzable by a single prompt. We then systematically
create a diverse set of simple, natural, and useful prompts to robustly analyze
each individual constraint. Using the GPT-3 text-davinci-002 model as a case
study, we generate outputs from our collection of prompts and analyze the
model's generative failures. We also show the generalizability of our proposed
method on other large models like BLOOM and OPT. Our results and our in-context
mitigation strategies reveal open challenges for future research. We have
publicly released our code at this https URL.

本文采用基于提示的方法，通过加入约束类型的多样性和可解析性，系统性地分析了开放式生成模型的限制及其失败原因，案例分析了 GPT-3 模型的生成失败，并揭示了未来研究的挑战。