As large language models (LLMs) expand the power of natural language
processing to handle long inputs, rigorous and systematic analyses are
necessary to understand their abilities and behavior. A salient application is
summarization, due to its ubiquity and controversy (e.g., researchers have
declared the death of summarization). In this paper, we use financial report
summarization as a case study because financial reports not only are long but
also use numbers and tables extensively. We propose a computational framework
for characterizing multimodal long-form summarization and investigate the
behavior of Claude 2.0/2.1, GPT-4/3.5, and Command. We find that GPT-3.5 and
Command fail to perform this summarization task meaningfully. For Claude 2 and
GPT-4, we analyze the extractiveness of the summary and identify a position
bias in LLMs. This position bias disappears after shuffling the input for
Claude, which suggests that Claude has the ability to recognize important
information. We also conduct a comprehensive investigation on the use of
numeric data in LLM-generated summaries and offer a taxonomy of numeric
hallucination. We employ prompt engineering to improve GPT-4's use of numbers
with limited success. Overall, our analyses highlight the strong capability of
Claude 2 in handling long multimodal inputs compared to GPT-4.

通过对财务报告摘要化的案例研究，我们提出了一种计算框架，用于表征多模态长文本摘要，并研究了 Claude 2.0/2.1、GPT-4/3.5 和 Command 的行为。我们发现 GPT-3.5 和 Command 无法有效地完成这个摘要任务，而 Claude 2 和 GPT-4 在摘要的提取性上存在偏差。随机打乱输入后，Claude 的位置偏见消失，这表明 Claude 有识别重要信息的能力。我们还对 LLM 生成的摘要中的数量数据进行了全面调查，并提供了数量幻觉的分类。我们尝试通过提示工程来改善 GPT-4 对数字的使用，但成功有限。总体而言，我们的分析凸显出 Claude 2 在处理长的多模态输入方面的强大能力，相较于 GPT-4。