Very large language models (LLMs) perform extremely well on a spectrum of NLP
tasks in a zero-shot setting. However, little is known about their performance
on human-level NLP problems which rely on understanding psychological concepts,
such as assessing personality traits. In this work, we investigate the
zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users'
social media posts. Through a set of systematic experiments, we find that
zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA
for broad classification upon injecting knowledge about the trait in the
prompts. However, when prompted to provide fine-grained classification, its
performance drops to close to a simple most frequent class (MFC) baseline. We
further analyze where GPT-3 performs better, as well as worse, than a
pretrained lexical model, illustrating systematic errors that suggest ways to
improve LLMs on human-level NLP tasks.

本文探讨了 GPT-3 在用户社交媒体帖子上对 Big 5 个性格特质的零样本估计能力，并发现其在粗分类上表现与现有的预训练模型相近，但在细粒度分类下表现不如常见类别基准，同时分析了 GPT-3 和预训练词汇模型表现较好和较差的领域，提出了改进 LNM 在人类级 NLP 任务上的建议。