Large Language Models (LLMs) excel in natural language tasks but face
limitations due to static training datasets, resulting in outdated or
contextually shallow responses. Retrieval-Augmented Generation (RAG) addresses
this by integrating real-time external knowledge, enhancing model accuracy and
credibility, especially for knowledge-intensive tasks. However, RAG-enhanced
LLMs struggle with long contexts, causing them to "choke" on information
overload, compromising response quality. Recent RAG applications use
hierarchical data structures for storing documents, organized at various levels
of summarization and information density. In this context, we introduce HIRO
(Hierarchical Information Retrieval Optimization), a novel querying approach
for RAG applications using hierarchical structures for storing documents. HIRO
employs DFS-based recursive similarity score calculation and branch pruning to
minimize the context returned to the LLM without informational loss. HIRO
outperforms existing querying mechanisms on the NarrativeQA dataset by an
absolute performance gain of 10.85%.

使用分层信息检索优化（HIRO）的查询方法，通过使用层次化结构来存储文档，从而减少返回给 LLM 的上下文并减少信息损失，提高了模型的性能。

HIRO：分层信息检索优化

HIRO: Hierarchical Information Retrieval Optimization

Hierarchical reinforcement learning (HRL) is a promising approach to extend
traditional reinforcement learning (RL) methods to solve more complex tasks.
Yet, the majority of current HRL methods require careful task-specific design
and on-policy training, making them difficult to apply in real-world scenarios.
In this paper, we study how we can develop HRL algorithms that are general, in
that they do not make onerous additional assumptions beyond standard RL
algorithms, and efficient, in the sense that they can be used with modest
numbers of interaction samples, making them suitable for real-world problems
such as robotic control. For generality, we develop a scheme where lower-level
controllers are supervised with goals that are learned and proposed
automatically by the higher-level controllers. To address efficiency, we
propose to use off-policy experience for both higher and lower-level training.
This poses a considerable challenge, since changes to the lower-level behaviors
change the action space for the higher-level policy, and we introduce an
off-policy correction to remedy this challenge. This allows us to take
advantage of recent advances in off-policy model-free RL to learn both higher-
and lower-level policies using substantially fewer environment interactions
than on-policy algorithms. We term the resulting HRL agent HIRO and find that
it is generally applicable and highly sample-efficient. Our experiments show
that HIRO can be used to learn highly complex behaviors for simulated robots,
such as pushing objects and utilizing them to reach target locations, learning
from only a few million samples, equivalent to a few days of real-time
interaction. In comparisons with a number of prior HRL methods, we find that
our approach substantially outperforms previous state-of-the-art techniques.

本文研究如何构建通用且高效的层次强化学习算法，其中较低层的控制器通过自动学习和提出的目标来实现上级控制器的监督，并使用脱离策略的经验来提高效率。我们称此算法为 HIRO，并在模拟机器人上的实验中表现出高性能和高样本效率。