Large Language Models (LLMs) have showcased impressive capabilities in
handling straightforward programming tasks. However, their performance tends to
falter when confronted with more challenging programming problems. We observe
that conventional models often generate solutions as monolithic code blocks,
restricting their effectiveness in tackling intricate questions. To overcome
this limitation, we present Modular-of-Thought Coder (MoTCoder). We introduce a
pioneering framework for MoT instruction tuning, designed to promote the
decomposition of tasks into logical sub-tasks and sub-modules. Our
investigations reveal that, through the cultivation and utilization of
sub-modules, MoTCoder significantly improves both the modularity and
correctness of the generated solutions, leading to substantial relative pass@1
improvements of 12.9% on APPS and 9.43% on CodeContests. Our codes are
available at this https URL

通过引入模块思维编码器（MoTCoder）框架，本研究针对大型语言模型在处理编程问题时的性能下降问题进行了研究，并提出了一种模块化指导下的任务分解方法，通过培育和利用子模块，显著提高了生成解决方案的模块化程度和正确性，从而在 APPS 和 CodeContests 上相对 pass@1 指标分别提升了 12.9% 和 9.43%。

MoTCoder: 模块化思维提升大型语言模型用于具有挑战性的编程任务

MoTCoder: Elevating Large Language Models with Modular of Thought for  Challenging Programming Tasks

Large Language Models (LLMs) have already become quite proficient at solving
simpler programming tasks like those in HumanEval or MBPP benchmarks. However,
solving more complex and competitive programming tasks is still quite
challenging for these models - possibly due to their tendency to generate
solutions as monolithic code blocks instead of decomposing them into logical
sub-tasks and sub-modules. On the other hand, experienced programmers
instinctively write modularized code with abstraction for solving complex
tasks, often reusing previously developed modules. To address this gap, we
propose CodeChain, a novel framework for inference that elicits modularized
code generation through a chain of self-revisions, each being guided by some
representative sub-modules generated in previous iterations. Concretely,
CodeChain first instructs the LLM to generate modularized codes through
chain-of-thought prompting. Then it applies a chain of self-revisions by
iterating the two steps: 1) extracting and clustering the generated sub-modules
and selecting the cluster representatives as the more generic and re-usable
implementations, and 2) augmenting the original chain-of-thought prompt with
these selected module-implementations and instructing the LLM to re-generate
new modularized solutions. We find that by naturally encouraging the LLM to
reuse the previously developed and verified sub-modules, CodeChain can
significantly boost both modularity as well as correctness of the generated
solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on
CodeContests. It is shown to be effective on both OpenAI LLMs as well as
open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation
studies with different methods of prompting, number of clusters, model sizes,
program qualities, etc., to provide useful insights that underpin CodeChain's
success.

通过一系列自我修订实现模块化代码生成的 CodeChain 框架，在解决复杂编程任务方面取得显著提升。

CodeChain: 通过自修订的子模块链实现模块化代码生成

CodeChain: Towards Modular Code Generation Through Chain of  Self-revisions with Representative Sub-modules

Recent years have witnessed a remarkable success of large deep learning
models. However, training these models is challenging due to high computational
costs, painfully slow convergence, and overfitting issues. In this paper, we
present Deep Incubation, a novel approach that enables the efficient and
effective training of large models by dividing them into smaller sub-modules
that can be trained separately and assembled seamlessly. A key challenge for
implementing this idea is to ensure the compatibility of the independently
trained sub-modules. To address this issue, we first introduce a global, shared
meta model, which is leveraged to implicitly link all the modules together, and
can be designed as an extremely small network with negligible computational
overhead. Then we propose a module incubation algorithm, which trains each
sub-module to replace the corresponding component of the meta model and
accomplish a given learning task. Despite the simplicity, our approach
effectively encourages each sub-module to be aware of its role in the target
large model, such that the finally-learned sub-modules can collaborate with
each other smoothly after being assembled. Empirically, our method outperforms
end-to-end (E2E) training in terms of both final accuracy and training
efficiency. For example, on top of ViT-Huge, it improves the accuracy by 2.7%
on ImageNet or achieves similar performance with 4x less training time.
Notably, the gains are significant for downstream tasks as well (e.g., object
detection and image segmentation on COCO and ADE20K). Code is available at
this https URL.

本文提出 Deep Incubation 训练方法，将大型深度学习模型分为互相连接的子模块进行训练，并经过实验证明在训练效率和准确率方面优于 end-to-end 训练方法。