Leveraging the powerful generative capability of diffusion models (DMs) to
build decision-making agents has achieved extensive success. However, there is
still a demand for an easy-to-use and modularized open-source library that
offers customized and efficient development for DM-based decision-making
algorithms. In this work, we introduce CleanDiffuser, the first DM library
specifically designed for decision-making algorithms. By revisiting the roles
of DMs in the decision-making domain, we identify a set of essential
sub-modules that constitute the core of CleanDiffuser, allowing for the
implementation of various DM algorithms with simple and flexible building
blocks. To demonstrate the reliability and flexibility of CleanDiffuser, we
conduct comprehensive evaluations of various DM algorithms implemented with
CleanDiffuser across an extensive range of tasks. The analytical experiments
provide a wealth of valuable design choices and insights, reveal opportunities
and challenges, and lay a solid groundwork for future research. CleanDiffuser
will provide long-term support to the decision-making community, enhancing
reproducibility and fostering the development of more robust solutions. The
code and documentation of CleanDiffuser are open-sourced on the
this https URL

CleanDiffuser 是首个专为决策算法设计的扩散模型库，通过提供简单灵活的构建模块实现各种扩散模型算法，并进行全面评估，为未来研究打下坚实基础。

CleanDiffuser：一个易于使用的模块化扩散模型库在决策中的应用

CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models  in Decision Making

Large vision-language models (VLMs) fine-tuned on specialized visual
instruction-following data have exhibited impressive language reasoning
capabilities across various scenarios. However, this fine-tuning paradigm may
not be able to efficiently learn optimal decision-making agents in multi-step
goal-directed tasks from interactive environments. To address this challenge,
we propose an algorithmic framework that fine-tunes VLMs with reinforcement
learning (RL). Specifically, our framework provides a task description and then
prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM
to efficiently explore intermediate reasoning steps that lead to the final
text-based action. Next, the open-ended text output is parsed into an
executable action to interact with the environment to obtain goal-directed task
rewards. Finally, our framework uses these task rewards to fine-tune the entire
VLM with RL. Empirically, we demonstrate that our proposed framework enhances
the decision-making capabilities of VLM agents across various tasks, enabling
7b models to outperform commercial models such as GPT4-V or Gemini.
Furthermore, we find that CoT reasoning is a crucial component for performance
improvement, as removing the CoT reasoning results in a significant decrease in
the overall performance of our method.

使用强化学习对视觉语言模型进行微调，提出了一种算法框架来增强其决策能力，验证了连续思维推理的重要性，并展示了在各种任务中超越商业模型的性能。

通过强化学习将大型视觉语言模型细调为决策代理

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via  Reinforcement Learning

The generalization of decision-making agents encompasses two fundamental
elements: learning from past experiences and reasoning in novel contexts.
However, the predominant emphasis in most interactive environments is on
learning, often at the expense of complexity in reasoning. In this paper, we
introduce CivRealm, an environment inspired by the Civilization game.
Civilization's profound alignment with human history and society necessitates
sophisticated learning, while its ever-changing situations demand strong
reasoning to generalize. Particularly, CivRealm sets up an
imperfect-information general-sum game with a changing number of players; it
presents a plethora of complex features, challenging the agent to deal with
open-ended stochastic environments that require diplomacy and negotiation
skills. Within CivRealm, we provide interfaces for two typical agent types:
tensor-based agents that focus on learning, and language-based agents that
emphasize reasoning. To catalyze further research, we present initial results
for both paradigms. The canonical RL-based agents exhibit reasonable
performance in mini-games, whereas both RL- and LLM-based agents struggle to
make substantial progress in the full game. Overall, CivRealm stands as a
unique learning and reasoning challenge for decision-making agents. The code is
available at this https URL

通过 CivRealm 环境，本论文介绍了决策代理的学习和推理两个基本要素，以及在互动环境中学习和推理之间的平衡问题。