Code Large Language Models (Code LLMs) have been increasingly used by
developers to boost productivity, but they often generate vulnerable code.
Thus, there is an urgent need to ensure that code generated by Code LLMs is
correct and secure. Previous research has primarily focused on generating
secure code, overlooking the fact that secure code also needs to be correct.
This oversight can lead to a false sense of security. Currently, the community
lacks a method to measure actual progress in this area, and we need solutions
that address both security and correctness of code generation.
This paper introduces a new benchmark, CodeGuard+, along with two new
metrics, secure-pass@k and secure@$k_{\text{pass}}$, to measure Code LLMs'
ability to generate both secure and correct code. Using our new evaluation
methods, we show that the state-of-the-art defense technique, prefix tuning,
may not be as strong as previously believed, since it generates secure code but
sacrifices functional correctness. We also demonstrate that different decoding
methods significantly affect the security of Code LLMs.
Furthermore, we explore a new defense direction: constrained decoding for
secure code generation. We propose new constrained decoding techniques to
generate code that satisfies security and correctness constraints
simultaneously. Our results reveal that constrained decoding is more effective
than prefix tuning to improve the security of Code LLMs, without requiring a
specialized training dataset. Moreover, constrained decoding can be used
together with prefix tuning to further improve the security of Code LLMs.

通过引入新的基准 CodeGuard + 和两个新的度量标准 secure-pass@k 和 secure@$k_{ext {pass}}$，本文展示了 Code Large Language Models 在生成安全和正确代码方面的能力，并指出了一种新的防御方向：约束解码技术，可以同时满足安全性和正确性要求。