Humans write code in a fundamentally interactive manner and rely on constant
execution feedback to correct errors, resolve ambiguities, and decompose tasks.
While LLMs have recently exhibited promising coding capabilities, current
coding benchmarks mostly consider a static instruction-to-code sequence
transduction process, which has the potential for error propagation and a
disconnect between the generated code and its final execution environment. To
address this gap, we introduce InterCode, a lightweight, flexible, and
easy-to-use framework of interactive coding as a standard reinforcement
learning (RL) environment, with code as actions and execution feedback as
observations. Our framework is language and platform agnostic, uses
self-contained Docker environments to provide safe and reproducible execution,
and is compatible out-of-the-box with traditional seq2seq coding methods, while
enabling the development of new methods for interactive code generation. We use
InterCode to create two interactive code environments with Bash and SQL as
action spaces, leveraging data from the static Spider and NL2Bash datasets. We
demonstrate InterCode's viability as a testbed by evaluating multiple
state-of-the-art LLMs configured with different prompting strategies such as
ReAct and Plan & Solve. Our results showcase the benefits of interactive code
generation and demonstrate that InterCode can serve as a challenging benchmark
for advancing code understanding and generation capabilities. InterCode is
designed to be easily extensible and can even be used to incorporate new tasks
such as Capture the Flag, a popular coding puzzle that is inherently multi-step
and involves multiple programming languages. Project site with code and data:
this https URL

本研究提出 InterCode 框架，作为交互式编码的标准强化学习（RL）环境，并使用 Bash 和 SQL 作为行动空间，以前沿的 LLMs 和其它编码策略作为实证对象，以展示其优越性和可持续性，同时证明其作为推进代码理解和生成能力的有挑战性的基准测试。