Understanding context is key to understanding human language, an ability
which Large Language Models (LLMs) have been increasingly seen to demonstrate
to an impressive extent. However, though the evaluation of LLMs encompasses
various domains within the realm of Natural Language Processing, limited
attention has been paid to probing their linguistic capability of understanding
contextual features. This paper introduces a context understanding benchmark by
adapting existing datasets to suit the evaluation of generative models. This
benchmark comprises of four distinct tasks and nine datasets, all featuring
prompts designed to assess the models' ability to understand context. First, we
evaluate the performance of LLMs under the in-context learning pretraining
scenario. Experimental results indicate that pre-trained dense models struggle
with understanding more nuanced contextual features when compared to
state-of-the-art fine-tuned models. Second, as LLM compression holds growing
significance in both research and real-world applications, we assess the
context understanding of quantized models under in-context-learning settings.
We find that 3-bit post-training quantization leads to varying degrees of
performance reduction on our benchmark. We conduct an extensive analysis of
these scenarios to substantiate our experimental results.

理解上下文是理解人类语言的关键，本论文介绍了一个上下文理解基准，通过适应现有数据集来评估生成模型的能力，并发现预先训练的稠密模型在理解复杂上下文特征方面比优化调整模型困难。