Classic reinforcement learning (RL) frequently confronts challenges in tasks
involving delays, which cause a mismatch between received observations and
subsequent actions, thereby deviating from the Markov assumption. Existing
methods usually tackle this issue with end-to-end solutions using state
augmentation. However, these black-box approaches often involve
incomprehensible processes and redundant information in the information states,
causing instability and potentially undermining the overall performance. To
alleviate the delay challenges in RL, we propose $\textbf{DEER (Delay-resilient
Encoder-Enhanced RL)}$, a framework designed to effectively enhance the
interpretability and address the random delay issues. DEER employs a pretrained
encoder to map delayed states, along with their variable-length past action
sequences resulting from different delays, into hidden states, which is trained
on delay-free environment datasets. In a variety of delayed scenarios, the
trained encoder can seamlessly integrate with standard RL algorithms without
requiring additional modifications and enhance the delay-solving capability by
simply adapting the input dimension of the original algorithms. We evaluate
DEER through extensive experiments on Gym and Mujoco environments. The results
confirm that DEER is superior to state-of-the-art RL algorithms in both
constant and random delay settings.

提出了一种名为 DEER（Delay-resilient Encoder-Enhanced RL）的框架，用于增强可解释性并解决随机延迟问题，通过使用预训练编码器将延迟状态及其不同延迟导致的可变长度的过去动作序列映射到隐藏状态，从而有效地缓解了强化学习中的延迟挑战。在延迟情境下，训练好的编码器可以与标准强化学习算法无缝集成，并通过适应原始算法的输入维度来增强解决延迟问题的能力。通过在 Gym 和 Mujoco 环境上进行广泛实验，结果证明 DEER 在固定和随机延迟设置下优于现有的强化学习算法。

DEER: 变延迟强化学习的抗延迟框架

DEER: A Delay-Resilient Framework for Reinforcement Learning with  Variable Delays

Tool-augmented large language models (LLMs) are attracting widespread
attention when accessing up-to-date knowledge and alleviating hallucination
issues. Nowadays, advanced closed-source LLMs (e.g., ChatGPT) have demonstrated
surprising tool-usage capabilities through prompting and in-context learning
techniques. To empower the capabilities of open-source LLMs (e.g., LLaMA) in
manipulating tools, current efforts focus on either template-driven or
token-triggered tool-usage. However, the former hampers LLMs' flexibility to
address diverse user's queries due to constrained tool interactions, while the
latter limits the generalizability when engaging with new tools, since
tool-usage learning is based on task- and tool-specific datasets. To alleviate
these concerns, in this paper, we propose a decision-aware and generalizable
tool-usage framework (DEER). Specifically, we first construct the tool-usage
samples with multiple decision branches via an automatic generation pipeline,
thereby inspiring the decision-making awareness of LLMs under diverse
scenarios. Meanwhile, we propose a novel tool sampling strategy to enhance the
generalizability of LLMs over unseen tools. Extensive experiments demonstrate
that our proposed DEER is effective and significantly outperforms baselines
across various datasets.

通过提出一种决策感知和具有一般化的工具使用框架（DEER），本文旨在增强开源 LLMs 在操作工具方面的能力。通过构建工具使用样本和新颖的工具采样策略，DEER 在各个数据集上证明了其有效性和显著优于基准模型。

着眼决策感知和通用化的大规模语言模型工具使用

Look Before You Leap: Towards Decision-Aware and Generalizable  Tool-Usage for Large Language Models

We propose DEER (Descriptive Knowledge Graph for Explaining Entity
Relationships) - an open and informative form of modeling entity relationships.
In DEER, relationships between entities are represented by free-text relation
descriptions. For instance, the relationship between entities of machine
learning and algorithm can be represented as ``Machine learning explores the
study and construction of algorithms that can learn from and make predictions
on data.'' To construct DEER, we propose a self-supervised learning method to
extract relation descriptions with the analysis of dependency patterns and
generate relation descriptions with a transformer-based relation description
synthesizing model, where no human labeling is required. Experiments
demonstrate that our system can extract and generate high-quality relation
descriptions for explaining entity relationships. The results suggest that we
can build an open and informative knowledge graph without human annotation.

本文提出了 DEER（用于解释实体关系的描述性知识图谱），采用一种自我监督学习的方法来提取关系描述，并使用基于转换的合成模型生成关系描述，最终成功构建了一个开放而丰富的知识图谱