We introduce GECKO, a bilingual large language model (LLM) optimized for
Korean and English, along with programming languages. GECKO is pretrained on
the balanced, high-quality corpus of Korean and English employing LLaMA
architecture. In this report, we share the experiences of several efforts to
build a better data pipeline for the corpus and to train our model. GECKO shows
great efficiency in token generations for both Korean and English, despite its
small size of vocabulary. We measure the performance on the representative
benchmarks in terms of Korean, English and Code, and it exhibits great
performance on KMMLU (Korean MMLU) and modest performance in English and Code,
even with its smaller number of trained tokens compared to English-focused
LLMs. GECKO is available to the open-source community under a permissive
license. We hope our work offers a research baseline and practical insights for
Korean LLM research. The model can be found at:
this https URL

GECKO 是一种针对韩语和英语以及编程语言进行优化的双语大型语言模型 (LLM)，通过 LLaMA 架构在平衡、高质量的韩语和英语语料库上进行预训练。该模型在韩语和英语的令牌生成效率上表现出色，尽管其词汇量较小。希望此研究能为韩语 LLM 研究提供一个基准和实际见解。

GECKO: 英文、程式碼和韓文的生成語言模型

GECKO: Generative Language Model for English, Code and Korean

We present Gecko, a compact and versatile text embedding model. Gecko
achieves strong retrieval performance by leveraging a key idea: distilling
knowledge from large language models (LLMs) into a retriever. Our two-step
distillation process begins with generating diverse, synthetic paired data
using an LLM. Next, we further refine the data quality by retrieving a set of
candidate passages for each query, and relabeling the positive and hard
negative passages using the same LLM. The effectiveness of our approach is
demonstrated by the compactness of the Gecko. On the Massive Text Embedding
Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
an average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.

我们提出了一种紧凑而多功能的文本嵌入模型 Gecko，其通过利用大语言模型（LLMs）将知识从 LLMs 中提炼到检索器中来实现强大的检索性能。