The inference phase of Large Language Models (LLMs) is very expensive. An
ideal inference stage of LLMs could utilize fewer computational resources while
still maintaining its capabilities (e.g., generalization and in-context
learning ability). In this paper, we try to answer the question, "During LLM
inference, can we use shallow layers for easy instances; and deep layers for
hard ones?" To answer this question, we first indicate that Not all Layers are
Necessary during Inference by statistically analyzing the activated layers
across tasks. Then, we propose a simple algorithm named AdaInfer to determine
the inference termination moment based on the input instance adaptively. More
importantly, AdaInfer does not alter LLM parameters and maintains
generalizability across tasks. Experiments on well-known LLMs (i.e., Llama2
series and OPT) show that AdaInfer saves an average of 14.8% of computational
resources, even up to 50% on sentiment tasks, while maintaining comparable
performance. Additionally, this method is orthogonal to other model
acceleration techniques, potentially boosting inference efficiency further.

利用适应性输入实例的简单算法 AdaInfer，在 Large Language Models 的推理阶段中使用浅层次进行简单实例的推理和深层次进行困难实例的推理，可在节省计算资源的同时保持性能。

推理过程中不需要所有 LLM 层

Not all Layers of LLMs are Necessary during Inference

Graph neural networks (GNNs) are one of the most rapidly growing fields
within deep learning. According to the growth in the dataset and the model size
used for GNNs, an important problem is that it becomes nearly impossible to
keep the whole network on GPU memory. Among numerous attempts, distributed
training is one popular approach to address the problem. However, due to the
nature of GNNs, existing distributed approaches suffer from poor scalability,
mainly due to the slow external server communications.
In this paper, we propose GraNNDis, an efficient distributed GNN training
framework for training GNNs on large graphs and deep layers. GraNNDis
introduces three new techniques. First, shared preloading provides a training
structure for a cluster of multi-GPU servers. We suggest server-wise preloading
of essential vertex dependencies to reduce the low-bandwidth external server
communications. Second, we present expansion-aware sampling. Because shared
preloading alone has limitations because of the neighbor explosion,
expansion-aware sampling reduces vertex dependencies that span across server
boundaries. Third, we propose cooperative batching to create a unified
framework for full-graph and minibatch training. It significantly reduces
redundant memory usage in mini-batch training. From this, GraNNDis enables a
reasonable trade-off between full-graph and mini-batch training through
unification especially when the entire graph does not fit into the GPU memory.
With experiments conducted on a multi-server/multi-GPU cluster, we show that
GraNNDis provides superior speedup over the state-of-the-art distributed GNN
training frameworks.

GraNNDis 是一个高效的分布式 GNN 训练框架，通过共享预加载、扩展感知采样和合作批处理三种技术，解决了大图和深层训练中的通信和内存问题，实现了超越现有分布式 GNN 训练框架的速度提升。

GraNNDis: 面向大规模集群的高效统一分布式训练框架用于深度图神经网络

GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs  on Large Clusters

We introduce a framework for analyzing various types of information in an NLP
Transformer. In this approach, we distinguish four layers of information:
positional, syntactic, semantic, and contextual. We also argue that the common
practice of adding positional information to semantic embedding is sub-optimal
and propose instead a Linear-and-Add approach. Our analysis reveals an
autogenetic separation of positional information through the deep layers. We
show that the distilled positional components of the embedding vectors follow
the path of a helix, both on the encoder side and on the decoder side. We
additionally show that on the encoder side, the conceptual dimensions generate
Part-of-Speech (PoS) clusters. On the decoder side, we show that a di-gram
approach helps to reveal the PoS clusters of the next token. Our approach paves
a way to elucidate the processing of information through the deep layers of an
NLP Transformer.

本研究提出了通过分析 NLP Transformer 中的不同信息类型的框架，并通过 Linear-and-Add 方法优化位置信息嵌入，发现位置信息具有深度分离特征，进而证明其对于 encoder 处的词性聚类和 decoder 处的下一个 token 的词性聚类具有启发意义。

NLP Transformer 中的双螺旋结构

The Double Helix inside the NLP Transformer

This paper aims to interpret the mechanism of feedforward ReLU networks by
exploring their solutions for piecewise linear functions, through the deduction
from basic rules. The constructed solution should be universal enough to
explain some network architectures of engineering; in order for that, several
ways are provided to enhance the solution universality. Some of the
consequences of our theories include: Under affine-geometry background, the
solutions of both three-layer networks and deep-layer networks are given,
particularly for those architectures applied in practice, such as multilayer
feedforward neural networks and decoders; We give clear and intuitive
interpretations of each component of network architectures; The
parameter-sharing mechanism for multi-outputs is investigated; We provide an
explanation of overparameterization solutions in terms of affine transforms;
Under our framework, an advantage of deep layers compared to shallower ones is
natural to be obtained. Some intermediate results are the basic knowledge for
the modeling or understanding of neural networks, such as the classification of
data embedded in a higher-dimensional space, the generalization of affine
transforms, the probabilistic model of matrix ranks, and the concept of
distinguishable data sets.

本文通过推导基本规则来解释前馈 ReLU 网络的机制，以解释工程网络架构，并提供了加强解决方案普遍性的几种方法。