Recent studies have demonstrated the potential to control paraphrase
generation, such as through syntax, which has broad applications in various
downstream tasks. However, these methods often require detailed parse trees or
syntactic exemplars, which are not user-friendly. Furthermore, an inference gap
exists, as control specifications are only available during training but not
inference. In this work, we propose a new setup for controlled paraphrasing.
Specifically, we represent user-intended actions as action tokens, allowing
embedding and concatenating them with text embeddings, thus flowing together to
a self-attention encoder for representation fusion. To address the inference
gap, we introduce an optional action token as a placeholder that encourages the
model to determine the appropriate action when control specifications are
inaccessible. Experimental results show that our method successfully enables
specific action-controlled paraphrasing and preserves the same or even better
performance compared to conventional uncontrolled methods when actions are not
given. Our findings thus promote the concept of optional action control for a
more user-centered design via representation learning.

通过引入操作令牌作为用户预期行为的表示，结合文本嵌入进行表示融合，以实现受控改写；实验结果表明，在不给定操作时，我们的方法能够成功实现特定操作受控的改写，并与传统的非受控方法保持相同甚至更好的性能，从而促进了面向用户的可选操作控制概念。

行动控制的改写

Action Controlled Paraphrasing

Human beings possess the capability to multiply a melange of multisensory
cues while actively exploring and interacting with the 3D world. Current
multi-modal large language models, however, passively absorb sensory data as
inputs, lacking the capacity to actively interact with the objects in the 3D
environment and dynamically collect their multisensory information. To usher in
the study of this area, we propose MultiPLY, a multisensory embodied large
language model that could incorporate multisensory interactive data, including
visual, audio, tactile, and thermal information into large language models,
thereby establishing the correlation among words, actions, and percepts. To
this end, we first collect Multisensory Universe, a large-scale multisensory
interaction dataset comprising 500k data by deploying an LLM-powered embodied
agent to engage with the 3D environment. To perform instruction tuning with
pre-trained LLM on such generated data, we first encode the 3D scene as
abstracted object-centric representations and then introduce action tokens
denoting that the embodied agent takes certain actions within the environment,
as well as state tokens that represent the multisensory state observations of
the agent at each time step. In the inference time, MultiPLY could generate
action tokens, instructing the agent to take the action in the environment and
obtain the next multisensory state observation. The observation is then
appended back to the LLM via state tokens to generate subsequent text or action
tokens. We demonstrate that MultiPLY outperforms baselines by a large margin
through a diverse set of embodied tasks involving object retrieval, tool use,
multisensory captioning, and task decomposition.

为了研究多感官交互和语言模型之间的相关性，我们提出了 MultiPLY，一个多感官体验的大型语言模型。通过将多感官交互数据引入大型语言模型，包括视觉、听觉、触觉和热觉信息，我们能够建立单词、行为和感知之间的关联。

MultiPLY：一个多感官，以物体为中心的基于体验的大型语言模型在 3D 世界中

MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in  3D World

We address the task of supervised action segmentation which aims to partition
a video into non-overlapping segments, each representing a different action.
Recent works apply transformers to perform temporal modeling at the
frame-level, which suffer from high computational cost and cannot well capture
action dependencies over long temporal horizons. To address these issues, we
propose an efficient BI-level Temporal modeling (BIT) framework that learns
explicit action tokens to represent action segments, in parallel performs
temporal modeling on frame and action levels, while maintaining a low
computational cost. Our model contains (i) a frame branch that uses convolution
to learn frame-level relationships, (ii) an action branch that uses transformer
to learn action-level dependencies with a small set of action tokens and (iii)
cross-attentions to allow communication between the two branches. We apply and
extend a set-prediction objective to allow each action token to represent one
or multiple action segments, thus can avoid learning a large number of tokens
over long videos with many segments. Thanks to the design of our action branch,
we can also seamlessly leverage textual transcripts of videos (when available)
to help action segmentation by using them to initialize the action tokens. We
evaluate our model on four video datasets (two egocentric and two third-person)
for action segmentation with and without transcripts, showing that BIT
significantly improves the state-of-the-art accuracy with much lower
computational cost (30 times faster) compared to existing transformer-based
methods.

我们提出了一种高效的 BI-level Temporal modeling (BIT) 框架，用于监督行动分割任务，通过学习明确的行动标记来表示行动段，在帧级和行动级上进行时间建模，同时保持较低的计算成本，提高了现有基于 transformer 的方法在行动分割方面的准确性和计算效率。