Collective Perception has attracted significant attention in recent years due
to its advantage for mitigating occlusion and expanding the field-of-view,
thereby enhancing reliability, efficiency, and, most crucially, decision-making
safety. However, developing collective perception models is highly resource
demanding due to extensive requirements of processing input data for many
agents, usually dozens of images and point clouds for a single frame. This not
only slows down the model development process for collective perception but
also impedes the utilization of larger models. In this paper, we propose an
agent-based training framework that handles the deep learning modules and agent
data separately to have a cleaner data flow structure. This framework not only
provides an API for flexibly prototyping the data processing pipeline and
defining the gradient calculation for each agent, but also provides the user
interface for interactive training, testing and data visualization. Training
experiment results of four collective object detection models on the prominent
collective perception benchmark OPV2V show that the agent-based training can
significantly reduce the GPU memory consumption and training time while
retaining inference performance. The framework and model implementations are
available at https://github.com/YuanYunshuang/CoSense3D

由于有效减少遮挡并扩展视野，群体感知近年来引起了广泛关注，从而提高可靠性、效率和决策安全性。本文提出了一种基于代理的训练框架，通过将深度学习模块和代理数据分开处理，实现了更清晰的数据流结构，既提供了用于灵活原型化数据处理流程和定义每个代理的梯度计算的 API，又提供了与用户交互式训练、测试和数据可视化的界面。在突出的群体感知基准 OPV2V 上，通过四个群体目标检测模型的训练实验结果表明，基于代理的训练可以显著减少 GPU 内存消耗和训练时间，并保持推理性能。

CoSense3D：一种基于智能体的高效学习框架用于集体感知

CoSense3D: an Agent-based Efficient Learning Framework for Collective  Perception

Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable
generalization capabilities to downstream tasks. However, existing prompt
tuning based frameworks need to parallelize learnable textual inputs for all
categories, suffering from massive GPU memory consumption when there is a large
number of categories in the target dataset. Moreover, previous works require to
include category names within prompts, exhibiting subpar performance when
dealing with ambiguous category names. To address these shortcomings, we
propose Compound Text-Guided Prompt Tuning (TGP-T) that significantly reduces
resource demand while achieving superior performance. We introduce text
supervision to the optimization of prompts, which enables two benefits: 1)
releasing the model reliance on the pre-defined category names during
inference, thereby enabling more flexible prompt generation; 2) reducing the
number of inputs to the text encoder, which decreases GPU memory consumption
significantly. Specifically, we found that compound text supervisions, i.e.,
category-wise and content-wise, is highly effective, since they provide
inter-class separability and capture intra-class variations, respectively.
Moreover, we condition the prompt generation on visual features through a
module called Bonder, which facilitates the alignment between prompts and
visual features. Extensive experiments on few-shot recognition and domain
generalization demonstrate that TGP-T achieves superior performance with
consistently lower training costs. It reduces GPU memory usage by 93% and
attains a 2.5% performance gain on 16-shot ImageNet. The code is available at
this https URL

对视觉语言模型进行复合文本引导的提示调整 (TGP-T)，通过引入文本监督和视觉特征对齐模块 (Bonder)，可以显著降低 GPU 内存占用，同时实现了卓越的性能。

通过图像适应提示调整的复合文本引导

Compound Text-Guided Prompt Tuning via Image-Adaptive Cues

Because of its streaming nature, recurrent neural network transducer (RNN-T)
is a very promising end-to-end (E2E) model that may replace the popular hybrid
model for automatic speech recognition. In this paper, we describe our recent
development of RNN-T models with reduced GPU memory consumption during
training, better initialization strategy, and advanced encoder modeling with
future lookahead. When trained with Microsoft's 65 thousand hours of anonymized
training data, the developed RNN-T model surpasses a very well trained hybrid
model with both better recognition accuracy and lower latency. We further study
how to customize RNN-T models to a new domain, which is important for deploying
E2E models to practical scenarios. By comparing several methods leveraging
text-only data in the new domain, we found that updating RNN-T's prediction and
joint networks using text-to-speech generated from domain-specific text is the
most effective.

本文中，我们介绍了最近开发的 RNN-T 模型，它在训练期间具有较小的 GPU 内存消耗、更好的初始化策略和先进的编码器建模，对 Microsoft 的 6.5 万小时的匿名训练数据进行训练后，开发的 RNN-T 模型优于经过训练良好的混合模型，同时具有更好的识别准确性和更低的延迟。我们进一步研究了如何将 RNN-T 模型定制为新领域，并比较了几种使用新领域纯文本数据的方法。研究发现，利用特定领域文本生成的文本到语音更新 RNN-T 的预测和联合网络最有效。