User alignment is crucial for adapting general-purpose language models (LMs)
to downstream tasks, but human annotations are often not available for all
types of instructions, especially those with customized constraints. We observe
that user instructions typically contain constraints. While assessing response
quality in terms of the whole instruction is often costly, efficiently
evaluating the satisfaction rate of constraints is feasible. We investigate
common constraints in NLP tasks, categorize them into three classes based on
the types of their arguments, and propose a unified framework, ACT (Aligning to
ConsTraints), to automatically produce supervision signals for user alignment
with constraints. Specifically, ACT uses constraint verifiers, which are
typically easy to implement in practice, to compute constraint satisfaction
rate (CSR) of each response. It samples multiple responses for each prompt and
collect preference labels based on their CSR automatically. Subsequently, ACT
adapts the LM to the target task through a ranking-based learning process.
Experiments on fine-grained entity typing, abstractive summarization, and
temporal question answering show that ACT is able to enhance LMs' capability to
adhere to different classes of constraints, thereby improving task performance.
Further experiments show that the constraint-following capabilities are
transferable.

使用者对齐是将通用语言模型（LMs）调整为下游任务所必不可少的，但人工注释往往无法涵盖所有类型的指令，尤其是具有定制约束的指令。本研究提出了一种统一框架 ACT（Aligning to ConsTraints），通过约束验证器自动产生用户对齐的监督信号，从而有效评估不同类别约束的满足率，并通过基于排序的学习过程来适应目标任务，实验证明 ACT 能够提高 LMs 对各种约束的遵循能力，从而改善任务性能，并且该约束遵循能力是可迁移的。

从指令到约束：与自动约束验证的语言模型对齐

From Instructions to Constraints: Language Model Alignment with  Automatic Constraint Verification

Neural Machine Translation (NMT) models have been shown to be vulnerable to
adversarial attacks, wherein carefully crafted perturbations of the input can
mislead the target model. In this paper, we introduce ACT, a novel adversarial
attack framework against NMT systems guided by a classifier. In our attack, the
adversary aims to craft meaning-preserving adversarial examples whose
translations by the NMT model belong to a different class than the original
translations in the target language. Unlike previous attacks, our new approach
has a more substantial effect on the translation by altering the overall
meaning, which leads to a different class determined by a classifier. To
evaluate the robustness of NMT models to this attack, we propose enhancements
to existing black-box word-replacement-based attacks by incorporating output
translations of the target NMT model and the output logits of a classifier
within the attack process. Extensive experiments in various settings, including
a comparison with existing untargeted attacks, demonstrate that the proposed
attack is considerably more successful in altering the class of the output
translation and has more effect on the translation. This new paradigm can show
the vulnerabilities of NMT systems by focusing on the class of translation
rather than the mere translation quality as studied traditionally.

我们提出了 ACT，一种新的针对 NMT 系统的对抗性攻击框架，通过一个分类器对其进行指导，以改变翻译的类别而非仅仅翻译质量，该攻击具有更大的影响力。