Large Language Models (LLMs) have significantly advanced natural language
processing tasks such as machine translation, text generation, and sentiment
analysis. However, their large size, often consisting of billions of
parameters, poses challenges for storage, computation, and deployment,
particularly in resource-constrained environments like mobile devices and edge
computing platforms. Effective compression and quantization techniques are
crucial for addressing these issues, reducing memory footprint and
computational requirements without significantly compromising performance.
Traditional methods that uniformly map parameters to compressed spaces fail to
account for the uneven distribution of parameters, leading to substantial
accuracy loss. In this work, we propose Athena, a novel algorithm for efficient
block-wise post-training quantization of LLMs. Athena leverages Second-Order
Matrix Derivative Information to guide the quantization process using the
curvature information of the loss landscape. By grouping parameters by columns
or rows and iteratively optimizing the quantization process, Athena updates the
model parameters and Hessian matrix to achieve significant compression while
maintaining high accuracy. This makes Athena a practical solution for deploying
LLMs in various settings.

提出了 Athena 算法，通过使用损失函数的曲率信息，将参数分组并进行迭代优化量化过程，通过更新模型参数和 Hessian 矩阵，在保持高准确性的同时实现显著的压缩，从而成为在各种环境中部署大型语言模型的实用解决方案。

Athena：大规模语言模型的高效块级后训练量化方法，使用二阶矩阵导数信息

Athena: Efficient Block-Wise Post-Training Quantization for Large  Language Models Using Second-Order Matrix Derivative Information

Solving math word problems depends on how to articulate the problems, the
lens through which models view human linguistic expressions. Real-world
settings count on such a method even more due to the diverse practices of the
same mathematical operations. Earlier works constrain available thinking
processes by limited prediction strategies without considering their
significance in acquiring mathematical knowledge. We introduce Attention-based
THought Expansion Network Architecture (ATHENA) to tackle the challenges of
real-world practices by mimicking human thought expansion mechanisms in the
form of neural network propagation. A thought expansion recurrently generates
the candidates carrying the thoughts of possible math expressions driven from
the previous step and yields reasonable thoughts by selecting the valid
pathways to the goal. Our experiments show that ATHENA achieves a new
state-of-the-art stage toward the ideal model that is compelling in variant
questions even when the informativeness in training examples is restricted.

使用注意力机制的 THought Expansion Network Architecture (ATHENA) 模型通过模仿人类思维扩展机制中的神经网络传播形式，解决实际世界的数学问题挑战，并在训练示例信息受限的情况下，在各种问题中取得了新的最佳表现。

ATHENA: 数理推理中的思维扩展

ATHENA: Mathematical Reasoning with Thought Expansion

There has been extensive research on developing defense techniques against
adversarial attacks; however, they have been mainly designed for specific model
families or application domains, therefore, they cannot be easily extended.
Based on the design philosophy of ensemble of diverse weak defenses, we propose
ATHENA---a flexible and extensible framework for building generic yet effective
defenses against adversarial attacks. We have conducted a comprehensive
empirical study to evaluate several realizations of ATHENA with four threat
models including zero-knowledge, black-box, gray-box, and white-box. We also
explain (i) why diversity matters, (ii) the generality of the defense
framework, and (iii) the overhead costs incurred by ATHENA.

提出了一种面向攻击的通用防御框架 ATHENA，采用弱防御的多样性设计原则，通过多种实现在零知识、黑盒、灰盒和白盒四种威胁模型下的评估，证明其高效并具有可扩展性，同时也说明了其高效性的原因和相关开销。