This paper describes the DSBA submissions to the Prompting Large Language
Models as Explainable Metrics shared task, where systems were submitted to two
tracks: small and large summarization tracks. With advanced Large Language
Models (LLMs) such as GPT-4, evaluating the quality of Natural Language
Generation (NLG) has become increasingly paramount. Traditional
similarity-based metrics such as BLEU and ROUGE have shown to misalign with
human evaluation and are ill-suited for open-ended generation tasks. To address
this issue, we explore the potential capability of LLM-based metrics,
especially leveraging open-source LLMs. In this study, wide range of prompts
and prompting techniques are systematically analyzed with three approaches:
prompting strategy, score aggregation, and explainability. Our research focuses
on formulating effective prompt templates, determining the granularity of NLG
quality scores and assessing the impact of in-context examples on LLM-based
evaluation. Furthermore, three aggregation strategies are compared to identify
the most reliable method for aggregating NLG quality scores. To examine
explainability, we devise a strategy that generates rationales for the scores
and analyzes the characteristics of the explanation produced by the open-source
LLMs. Extensive experiments provide insights regarding evaluation capabilities
of open-source LLMs and suggest effective prompting strategies.

通过分析提示策略、评分聚合和解释性等多个方面，本研究探索了基于开源大型语言模型（LLMs）的度量方法在自然语言生成品质评估中的潜力，并提供了有关开源 LLMs 评估能力的实验结果和有效的提示策略。