We introduce PaLM 2, a new state-of-the-art language model that has better
multilingual and reasoning capabilities and is more compute-efficient than its
predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture
of objectives. Through extensive evaluations on English and multilingual
language, and reasoning tasks, we demonstrate that PaLM 2 has significantly
improved quality on downstream tasks across different model sizes, while
simultaneously exhibiting faster and more efficient inference compared to PaLM.
This improved efficiency enables broader deployment while also allowing the
model to respond faster, for a more natural pace of interaction. PaLM 2
demonstrates robust reasoning capabilities exemplified by large improvements
over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable
performance on a suite of responsible AI evaluations, and enables
inference-time control over toxicity without additional overhead or impact on
other capabilities. Overall, PaLM 2 achieves state-of-the-art performance
across a diverse set of tasks and capabilities.
When discussing the PaLM 2 family, it is important to distinguish between
pre-trained models (of various sizes), fine-tuned variants of these models, and
the user-facing products that use these models. In particular, user-facing
products typically include additional pre- and post-processing steps.
Additionally, the underlying models may evolve over time. Therefore, one should
not expect the performance of user-facing products to exactly match the results
reported in this report.

PaLM 2 是个新的最先进的语言模型，比其前身 PaLM 在多语言和推理能力方面更具优势，同时也更加计算效率高。通过在英语、多语言和推理任务上的全面评估，我们证明了 PaLM 2 在不同模型大小的下游任务上的质量显著提高，同时推理速度更快、更高效，这提高了交互的自然节奏。因此，PaLM 2 达到了各种任务和能力的最先进水平。

PaLM 2 技术报告

PaLM 2 Technical Report

Recently,there has been a lot of interest in building compact models for
video classification which have a small memory footprint (<1 GB). While these
models are compact, they typically operate by repeated application of a small
weight matrix to all the frames in a video. E.g. recurrent neural network based
methods compute a hidden state for every frame of the video using a recurrent
weight matrix. Similarly, cluster-and-aggregate based methods such as NetVLAD,
have a learnable clustering matrix which is used to assign soft-clusters to
every frame in the video. Since these models look at every frame in the video,
the number of floating point operations (FLOPs) is still large even though the
memory footprint is small. We focus on building compute-efficient video
classification models which process fewer frames and hence have less number of
FLOPs. Similar to memory efficient models, we use the idea of distillation
albeit in a different setting. Specifically, in our case, a compute-heavy
teacher which looks at all the frames in the video is used to train a
compute-efficient student which looks at only a small fraction of frames in the
video. This is in contrast to a typical memory efficient Teacher-Student
setting, wherein both the teacher and the student look at all the frames in the
video but the student has fewer parameters. Our work thus complements the
research on memory efficient video classification. We do an extensive
evaluation with three types of models for video classification,viz.(i)
recurrent models (ii) cluster-and-aggregate models and (iii) memory-efficient
cluster-and-aggregate models and show that in each of these cases, a see-it-all
teacher can be used to train a compute efficient see-very-little student. We
show that the proposed student network can reduce the inference time by 30% and
the number of FLOPs by approximately 90% with a negligible drop in the
performance.

探索了一种使用高计算复杂度的 Teacher 来训练计算效率更高的 Student 进行视频分类的方法，并表明该方法可以显著减少 FLOPs 数量并降低推理时间。