People tell lies when seeking rewards. Large language models (LLMs) are
aligned to human values with reinforcement learning where they get rewards if
they satisfy human preference. We find that this also induces dishonesty in
helpful and harmless alignment where LLMs tell lies in generating harmless
responses. Using the latest interpreting tools, we detect dishonesty, show how
LLMs can be harmful if their honesty is increased, and analyze such conflicts
at the parameter-level. Given these preliminaries and the hypothesis that
reward-seeking stimulates dishonesty, we theoretically show that the dishonesty
can in-turn decrease the alignment performances and augment reward-seeking
alignment with representation regularization. Extensive results, including
GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we
can train more honest, helpful, and harmless LLMs. We will make all our codes
and results be open-sourced upon this paper's acceptance.

大规模语言模型在人类价值观上通过强化学习与人类价好矛盾时，也会导致诚实度降低，但通过进行特征规范化可以增强奖励导向的对齐。经过丰富的实验结果表明，我们可以训练出更加诚实、有帮助且无害的大规模语言模型。