Code generation aims to understand the problem description and generate
corresponding code snippets, where existing works generally decompose such
complex tasks into intermediate steps by prompting strategies, such as
Chain-of-Thought and its variants. While these studies have achieved some
success, their effectiveness is highly dependent on the capabilities of
advanced Large Language Models (LLMs) such as GPT-4, particularly in terms of
API calls, which significantly limits their practical applicability.
Consequently, how to enhance the code generation capabilities of small and
medium-scale code LLMs without significantly increasing training costs is an
appealing challenge. In this paper, we suggest that code comments are the
natural logic pivot between natural language and code language and propose
using comments to boost the code generation ability of code LLMs. Concretely,
we propose MANGO (comMents As Natural loGic pivOts), including a comment
contrastive training strategy and a corresponding logical comment decoding
strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder
and WizardCoder as backbone models, and encompassing model parameter sizes
between 3B and 7B. The results indicate that MANGO significantly improves the
code pass rate based on the strong baselines. Meanwhile, the robustness of the
logical comment decoding strategy is notably higher than the Chain-of-thoughts
prompting. The code is publicly available at
https://github.com/pppa2019/Mango.

借助注释作为代码 LLMs 生成能力的增强策略，MANGO 显著提高了代码通过率，并且逻辑注释解码策略的鲁棒性明显高于思路链引导。

自然逻辑枢轴的评论：通过评论角度改进代码生成

Comments as Natural Logic Pivots: Improve Code Generation via Comment  Perspective

Large language models such as ChatGPT and GPT-4 have recently achieved
astonishing performance on a variety of natural language processing tasks. In
this paper, we propose MANGO, a benchmark to evaluate their capabilities to
perform text-based mapping and navigation. Our benchmark includes 53 mazes
taken from a suite of textgames: each maze is paired with a walkthrough that
visits every location but does not cover all possible paths. The task is
question-answering: for each maze, a large language model reads the walkthrough
and answers hundreds of mapping and navigation questions such as "How should
you go to Attic from West of House?" and "Where are we if we go north and east
from Cellar?". Although these questions are easy to humans, it turns out that
even GPT-4, the best-to-date language model, performs poorly at answering them.
Further, our experiments suggest that a strong mapping and navigation ability
would benefit large language models in performing relevant downstream tasks,
such as playing textgames. Our MANGO benchmark will facilitate future research
on methods that improve the mapping and navigation capabilities of language
models. We host our leaderboard, data, code, and evaluation program at
this https URL and this https URL

提出了 MANGO 基准测试，用于评估大型语言模型在执行基于文本的映射和导航任务时的能力，结果表明即使是迄今为止最强的语言模型 GPT-4 在回答这些问题上表现不佳，而具备强大的映射和导航能力的语言模型对于执行相关下游任务（如玩文本游戏）将会有所好处。

MANGO：大型语言模型映射和导航能力评估基准

MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of  Large Language Models

Large-scale pre-trained multimodal transformers, such as ViLBERT and UNITER,
have propelled the state of the art in vision-and-language (V+L) research to a
new level. Although achieving impressive performance on standard tasks, to
date, it still remains unclear how robust these pre-trained models are. To
investigate, we conduct a host of thorough evaluations on existing pre-trained
models over 4 different types of V+L specific model robustness: (i) Linguistic
Variation; (ii) Logical Reasoning; (iii) Visual Content Manipulation; and (iv)
Answer Distribution Shift. Interestingly, by standard model finetuning,
pre-trained V+L models already exhibit better robustness than many
task-specific state-of-the-art methods. To further enhance model robustness, we
propose Mango, a generic and efficient approach that learns a Multimodal
Adversarial Noise GeneratOr in the embedding space to fool pre-trained V+L
models. Differing from previous studies focused on one specific type of
robustness, Mango is task-agnostic, and enables universal performance lift for
pre-trained models over diverse tasks designed to evaluate broad aspects of
robustness. Comprehensive experiments demonstrate that Mango achieves new state
of the art on 7 out of 9 robustness benchmarks, surpassing existing methods by
a significant margin. As the first comprehensive study on V+L robustness, this
work puts robustness of pre-trained models into sharper focus, pointing new
directions for future study.

通过对现有的预训练模型进行全面评估和改进，本研究提出了一种名为 Mango 的方法，在嵌入空间中学习多模态对抗性噪声生成器，使得预训练的视觉 - 语言模型的鲁棒性得到了大幅度提升，并在七项鲁棒性测试中创造了新的最高水平。