The inference phase of Large Language Models (LLMs) is very expensive. An
ideal inference stage of LLMs could utilize fewer computational resources while
still maintaining its capabilities (e.g., generalization and in-context
learning ability). In this paper, we try to answer the question, "During LLM
inference, can we use shallow layers for easy instances; and deep layers for
hard ones?" To answer this question, we first indicate that Not all Layers are
Necessary during Inference by statistically analyzing the activated layers
across tasks. Then, we propose a simple algorithm named AdaInfer to determine
the inference termination moment based on the input instance adaptively. More
importantly, AdaInfer does not alter LLM parameters and maintains
generalizability across tasks. Experiments on well-known LLMs (i.e., Llama2
series and OPT) show that AdaInfer saves an average of 14.8% of computational
resources, even up to 50% on sentiment tasks, while maintaining comparable
performance. Additionally, this method is orthogonal to other model
acceleration techniques, potentially boosting inference efficiency further.

利用适应性输入实例的简单算法 AdaInfer，在 Large Language Models 的推理阶段中使用浅层次进行简单实例的推理和深层次进行困难实例的推理，可在节省计算资源的同时保持性能。