In-context learning of large-language models (LLMs) has achieved remarkable
success in the field of natural language processing, while extensive case
studies reveal that the single-step chain-of-thought prompting approach faces
challenges such as attention diffusion and inadequate performance in complex
tasks like text-to-SQL. To improve the contextual learning capabilities of LLMs
in text-to-SQL, a workflow paradigm method is proposed, aiming to enhance the
attention and problem-solving scope of LLMs through decomposition.
Specifically, the information determination module for eliminating redundant
information and the brand-new prompt structure based on problem classification
greatly enhance the model's attention. Additionally, the inclusion of
self-correcting and active learning modules greatly expands the problem-solving
scope of LLMs, hence improving the upper limit of LLM-based approaches.
Extensive experiments conducted on three datasets demonstrate that our approach
outperforms other methods by a significant margin. About 2-3 percentage point
improvements compared to the existing baseline on the Spider Dev and
Spider-Realistic datasets and new SOTA results on the Spider Test dataset are
achieved. Our code is available on GitHub:
https://github.com/FlyingFeather/DEA-SQL.

通过工作流模式方法，改善大型语言模型在文本到 SQL 任务中的上下文学习能力，提高关注力和问题解决范围，包括信息确定模块、基于问题分类的全新提示结构，以及自我纠正和主动学习模块的引入。在三个数据集上的广泛实验表明，我们的方法表现优于其他方法，显著提高了基准指标，并在 Spider 测试数据集上取得了新的最佳结果。

通过工作流范式增强关注机制：改进基于 LLM 的文本到 SQL 的分解

Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL  through Workflow Paradigm

Efficient Transformers have been developed for long sequence modeling, due to
their subquadratic memory and time complexity. Sparse Transformer is a popular
approach to improving the efficiency of Transformers by restricting
self-attention to locations specified by the predefined sparse patterns.
However, leveraging sparsity may sacrifice expressiveness compared to
full-attention, when important token correlations are multiple hops away. To
combine advantages of both the efficiency of sparse transformer and the
expressiveness of full-attention Transformer, we propose \textit{Diffuser}, a
new state-of-the-art efficient Transformer. Diffuser incorporates all token
interactions within one attention layer while maintaining low computation and
memory costs. The key idea is to expand the receptive field of sparse attention
using Attention Diffusion, which computes multi-hop token correlations based on
all paths between corresponding disconnected tokens, besides attention among
neighboring tokens. Theoretically, we show the expressiveness of Diffuser as a
universal sequence approximator for sequence-to-sequence modeling, and
investigate its ability to approximate full-attention by analyzing the graph
expander property from the spectral perspective. Experimentally, we investigate
the effectiveness of Diffuser with extensive evaluations, including language
modeling, image modeling, and Long Range Arena (LRA). Evaluation results show
that Diffuser achieves improvements by an average of 0.94% on text
classification tasks and 2.30% on LRA, with 1.67$\times$ memory savings
compared to state-of-the-art benchmarks, which demonstrates superior
performance of Diffuser in both expressiveness and efficiency aspects.

提出了一种新的高效 Transformer 模型 Diffuser，在保持低计算量和内存成本的同时，通过 Attention Diffusion 扩大了基于稀疏注意力的 receptive field，并且通过实验证明，在文本分类和 LRA 等任务中，相比于现有的 benchmark 方法，Diffuser 可以平均提高 0.94％和 2.30％，可以显著提高效率和表现。