Large language models (LLMs) have rapidly evolved as the foundation of
various natural language processing (NLP) applications. Despite their wide use
cases, their understanding of culturally-related concepts and reasoning remains
limited. Meantime, there is a significant need to enhance these models'
cultural reasoning capabilities, especially concerning underrepresented
regions. This paper introduces a novel pipeline for extracting high-quality,
culturally-related instruction tuning datasets from vast unstructured corpora.
We utilize a self-instruction generation pipeline to identify cultural concepts
and trigger instruction. By integrating with a general-purpose instruction
tuning dataset, our model demonstrates enhanced capabilities in recognizing and
understanding regional cultural nuances, thereby enhancing its reasoning
capabilities. We conduct experiments across three regions: Singapore, the
Philippines, and the United States, achieving performance improvement of up to
6%. Our research opens new avenues for extracting cultural instruction tuning
sets directly from unstructured data, setting a precedent for future
innovations in the field.

本文介绍了一种从庞大的非结构化语料库中提取高质量文化相关调优数据集的新型流程。通过自我生成流程识别文化概念和触发指令，并与通用指令调优数据集相结合，我们的模型展示出了较强的识别和理解区域文化细微差别的能力，从而增强了其推理能力。我们在新加坡、菲律宾和美国三个地区开展了实验，取得了高达 6% 的性能改进。我们的研究为直接从非结构化数据中提取文化指令调优集开辟了新的途径，并为未来在该领域的创新设立了先例。