Recent developments in large language models (LLMs) have shown promise in
enhancing the capabilities of natural language processing (NLP). Despite these
successes, there remains a dearth of research dedicated to the NLP
problem-solving abilities of LLMs. To fill the gap in this area, we present a
unique benchmarking dataset, NLPBench, comprising 378 college-level NLP
questions spanning various NLP topics sourced from Yale University's prior
final exams. NLPBench includes questions with context, in which multiple
sub-questions share the same public information, and diverse question types,
including multiple choice, short answer, and math. Our evaluation, centered on
LLMs such as GPT-3.5/4, PaLM-2, and LLAMA-2, incorporates advanced prompting
strategies like the chain-of-thought (CoT) and tree-of-thought (ToT). Our study
reveals that the effectiveness of the advanced prompting strategies can be
inconsistent, occasionally damaging LLM performance, especially in smaller
models like the LLAMA-2 (13b). Furthermore, our manual assessment illuminated
specific shortcomings in LLMs' scientific problem-solving skills, with
weaknesses in logical decomposition and reasoning notably affecting results.

通过独特的基准数据集 NLPBench，评估了大型语言模型在自然语言处理中的问题解决能力，并发现高级提示策略的有效性不稳定，对 LLMs 性能有时造成损害，尤其是较小的模型 LLAMA-2（13 亿参数）中表现更明显；同时发现大型语言模型在科学问题解决能力方面存在特定的不足，逻辑分解和推理的薄弱性明显影响结果。

NLPBench：评估大型语言模型在解决 NLP 问题上的能力

NLPBench: Evaluating Large Language Models on Solving NLP Problems

Recently, ChatGPT has attracted great attention, as it can generate fluent
and high-quality responses to human inquiries. Several prior studies have shown
that ChatGPT attains remarkable generation ability compared with existing
models. However, the quantitative analysis of ChatGPT's understanding ability
has been given little attention. In this report, we explore the understanding
ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and
comparing it with 4 representative fine-tuned BERT-style models. We find that:
1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT
outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT
achieves comparable performance compared with BERT on sentiment analysis and
question-answering tasks. Additionally, by combining some advanced prompting
strategies, we show that the understanding ability of ChatGPT can be further
improved.

本研究评估了 ChatGPT 对最流行的 GLUE 基准的理解能力，并与 4 个代表性的 fine-tuned 的 BERT 模型进行比较。我们发现，ChatGPT 在处理释义和相似性任务方面存在不足，但在推理任务方面优于所有 BERT 模型，并在情感分析和问答任务上表现与 BERT 相当。此外，通过组合一些高级提示策略，我们展示了 ChatGPT 的理解能力可以进一步提高。