Even though large language models (LLMs) have demonstrated remarkable
capability in solving various natural language tasks, the capability of an LLM
to follow human instructions is still a concern. Recent works have shown great
improvements in the instruction-following capability via additional training
for instruction-following tasks. However, the mechanisms responsible for
effective instruction-following capabilities remain inadequately understood.
Here, we introduce a simplified instruction-following task and use synthetic
datasets to analyze a Transformer-based causal language model. Our findings
suggest that the model learns task-specific information by clustering data
within its hidden space, with this clustering process evolving dynamically
during learning. We also demonstrate how this phenomenon assists the model in
handling unseen instances and validate our results in a more realistic setting.

通过对合成数据集的分析，我们发现大规模语言模型通过将数据在其隐藏空间内进行聚类学习特定任务的信息，并且这个聚类过程在学习过程中动态演化，从而帮助模型处理未知的实例。

基于 Transformer 的因果语言模型进行聚类

Transformer-based Causal Language Models Perform Clustering

Detecting unseen instances based on multi-view templates is a challenging
problem due to its open-world nature. Traditional methodologies, which
primarily rely on 2D representations and matching techniques, are often
inadequate in handling pose variations and occlusions. To solve this, we
introduce VoxDet, a pioneer 3D geometry-aware framework that fully utilizes the
strong 3D voxel representation and reliable voxel matching mechanism. VoxDet
first ingeniously proposes template voxel aggregation (TVA) module, effectively
transforming multi-view 2D images into 3D voxel features. By leveraging
associated camera poses, these features are aggregated into a compact 3D
template voxel. In novel instance detection, this voxel representation
demonstrates heightened resilience to occlusion and pose variations. We also
discover that a 3D reconstruction objective helps to pre-train the 2D-3D
mapping in TVA. Second, to quickly align with the template voxel, VoxDet
incorporates a Query Voxel Matching (QVM) module. The 2D queries are first
converted into their voxel representation with the learned 2D-3D mapping. We
find that since the 3D voxel representations encode the geometry, we can first
estimate the relative rotation and then compare the aligned voxels, leading to
improved accuracy and efficiency. Exhaustive experiments are conducted on the
demanding LineMod-Occlusion, YCB-video, and the newly built RoboTools
benchmarks, where VoxDet outperforms various 2D baselines remarkably with 20%
higher recall and faster speed. To the best of our knowledge, VoxDet is the
first to incorporate implicit 3D knowledge for 2D tasks.

使用 VoxDet 这个基于 3D 几何感知的框架，通过模板体素聚合模块和查询体素匹配模块，将多视角 2D 图像转化为 3D 体素，从而实现对姿势变化和遮挡的有效识别，取得了 20% 较高的召回率和较快的速度。