Decoding using tree search can greatly enhance the inference quality for
transformer-based Large Language Models (LLMs). Depending on the guidance
signal, it searches for the best path from root to leaf in the tree by forming
LLM outputs to improve controllability, reasoning ability, alignment, et
cetera. However, current tree decoding strategies and their inference systems
do not suit each other well due to redundancy in computation, memory
footprints, and memory access, resulting in inefficient inference. To address
this issue, we propose DeFT, an IO-aware tree attention algorithm that
maintains memory-efficient attention calculation with low memory footprints in
two stages: (1) QKV Preparation: we propose a KV-Guided Tree Split strategy to
group QKV wisely for high utilization of GPUs and reduction of memory
reads/writes for the KV cache between GPU global memory and on-chip shared
memory as much as possible; (2) Attention Calculation: we calculate partial
attention of each QKV groups in a fused kernel then apply a Tree-topology-aware
Global Reduction strategy to get final attention. Thanks to a reduction in KV
cache IO by 3.6-4.5$\times$, along with an additional reduction in IO for
$\mathbf{Q} \mathbf{K}^\top$ and Softmax equivalent to 25% of the total KV
cache IO, DeFT can achieve a speedup of 1.7-2.4$\times$ in end-to-end latency
across two practical reasoning tasks over the SOTA attention algorithms.

通过树搜索解码，DeFT 提出了一种基于 IO 的树注意力算法，实现了内存高效的注意力计算，加速了端到端推理任务的运行速度。

DeFT: 带有 IO 感知的快速树注意力的树搜索型 LLM 推理

DeFT: Flash Tree-attention with IO-Awareness for Efficient  Tree-search-based LLM Inference

Dexterity is often seen as a cornerstone of complex manipulation. Humans are
able to perform a host of skills with their hands, from making food to
operating tools. In this paper, we investigate these challenges, especially in
the case of soft, deformable objects as well as complex, relatively
long-horizon tasks. However, learning such behaviors from scratch can be data
inefficient. To circumvent this, we propose a novel approach, DEFT (DExterous
Fine-Tuning for Hand Policies), that leverages human-driven priors, which are
executed directly in the real world. In order to improve upon these priors,
DEFT involves an efficient online optimization procedure. With the integration
of human-based learning and online fine-tuning, coupled with a soft robotic
hand, DEFT demonstrates success across various tasks, establishing a robust,
data-efficient pathway toward general dexterous manipulation. Please see our
website at this https URL for video results.

通过 DEFT（DExterous Fine-Tuning for Hand Policies）方法，结合软体机器手、基于人类驱动的经验和在线优化过程，本文探索了灵活性和复杂操作任务，并取得了成功的实验结果，展示了通向全面灵巧操作的数据高效路径。

DEFT: 灵巧微调用于现实世界手部策略

DEFT: Dexterous Fine-Tuning for Real-World Hand Policies

Gradient sparsification is a widely adopted solution for reducing the
excessive communication traffic in distributed deep learning. However, most
existing gradient sparsifiers have relatively poor scalability because of
considerable computational cost of gradient selection and/or increased
communication traffic owing to gradient build-up. To address these challenges,
we propose a novel gradient sparsification scheme, DEFT, that partitions the
gradient selection task into sub tasks and distributes them to workers. DEFT
differs from existing sparsifiers, wherein every worker selects gradients among
all gradients. Consequently, the computational cost can be reduced as the
number of workers increases. Moreover, gradient build-up can be eliminated
because DEFT allows workers to select gradients in partitions that are
non-intersecting (between workers). Therefore, even if the number of workers
increases, the communication traffic can be maintained as per user requirement.
To avoid the loss of significance of gradient selection, DEFT selects more
gradients in the layers that have a larger gradient norm than the other layers.
Because every layer has a different computational load, DEFT allocates layers
to workers using a bin-packing algorithm to maintain a balanced load of
gradient selection between workers. In our empirical evaluation, DEFT shows a
significant improvement in training performance in terms of speed in gradient
selection over existing sparsifiers while achieving high convergence
performance.

梯度稀疏化是减少分布式深度学习中过多通信流量的广泛采用解决方案，然而，大多数现有的梯度稀疏化方法由于梯度选择的计算成本较高和 / 或由于梯度累积导致的通信流量增加而具有相对较差的可扩展性。为了解决这些挑战，我们提出了一种新颖的梯度稀疏化方案 DEFT，它将梯度选择任务划分为子任务并将其分配给工作节点。DEFT 与现有的稀疏化方法不同，在于每个工作节点在所有梯度中选择梯度。因此，随着工作节点的增加，可以减少计算成本。此外，DEFT 可以消除梯度累积，因为允许工作节点在不重叠的分区中选择梯度。因此，即使工作节点的数量增加，也可以根据用户要求保持通信流量不变。为了避免梯度选择的重要性损失，DEFT 在梯度范数较大的层中选择更多的梯度。由于每个层的计算负载不同，DEFT 使用装箱算法将层分配给工作节点，以在工作节点之间保持平衡的梯度选择负载。在我们的实证评估中，DEFT 在梯度选择的速度方面相对于现有的稀疏化方法显示出显著的训练性能改进，同时实现了高收敛性能。