The pretrain-finetuning paradigm in large-scale sequence models has made
significant progress in natural language processing and computer vision tasks.
However, such a paradigm is still hindered by several challenges in
Reinforcement Learning (RL), including the lack of self-supervised pretraining
algorithms based on offline data and efficient fine-tuning/prompt-tuning over
unseen downstream tasks. In this work, we explore how prompts can improve
sequence modeling-based offline reinforcement learning (offline-RL) algorithms.
Firstly, we propose prompt tuning for offline RL, where a context vector
sequence is concatenated with the input to guide the conditional policy
generation. As such, we can pretrain a model on the offline dataset with
self-supervised loss and learn a prompt to guide the policy towards desired
actions. Secondly, we extend our framework to Meta-RL settings and propose
Contextual Meta Transformer (CMT); CMT leverages the context among different
tasks as the prompt to improve generalization on unseen tasks. We conduct
extensive experiments across three different offline-RL settings: offline
single-agent RL on the D4RL dataset, offline Meta-RL on the MuJoCo benchmark,
and offline MARL on the SMAC benchmark. Superior results validate the strong
performance, and generality of our methods.

本文探讨了如何通过 prompt tuning 和 Contextual Meta Transformer 算法来提高基于序列建模的离线强化学习算法的性能，并在三种不同的离线 RL 设置下进行了广泛的实验，验证了方法的高效性和普适性。

离线元强化学习的上下文变换器

Contextual Transformer for Offline Meta Reinforcement Learning

Recently, the pretrain-finetuning paradigm has attracted tons of attention in
graph learning community due to its power of alleviating the lack of labels
problem in many real-world applications. Current studies use existing
techniques, such as weight constraint, representation constraint, which are
derived from images or text data, to transfer the invariant knowledge from the
pre-train stage to fine-tuning stage. However, these methods failed to preserve
invariances from graph structure and Graph Neural Network (GNN) style models.
In this paper, we present a novel optimal transport-based fine-tuning framework
called GTOT-Tuning, namely, Graph Topology induced Optimal Transport
fine-Tuning, for GNN style backbones. GTOT-Tuning is required to utilize the
property of graph data to enhance the preservation of representation produced
by fine-tuned networks. Toward this goal, we formulate graph local knowledge
transfer as an Optimal Transport (OT) problem with a structural prior and
construct the GTOT regularizer to constrain the fine-tuned model behaviors. By
using the adjacency relationship amongst nodes, the GTOT regularizer achieves
node-level optimal transport procedures and reduces redundant transport
procedures, resulting in efficient knowledge transfer from the pre-trained
models. We evaluate GTOT-Tuning on eight downstream tasks with various GNN
backbones and demonstrate that it achieves state-of-the-art fine-tuning
performance for GNNs.

本研究提出了一种基于最优传输的微调框架，称为图拓扑诱导的最优传输微调（GTOT-Tuning），用于增强在图学习上预训练模型微调的表示保存性，并证明它在各种图神经网络模型上比现有技术表现更好。