With the primary focus on evaluating the effectiveness of large language
models for automatic reference-less translation assessment, this work presents
our experiments on mimicking human direct assessment to evaluate the quality of
translations in English and Indian languages. We constructed a translation
evaluation task where we performed zero-shot learning, in-context
example-driven learning, and fine-tuning of large language models to provide a
score out of 100, where 100 represents a perfect translation and 1 represents a
poor translation. We compared the performance of our trained systems with
existing methods such as COMET, BERT-Scorer, and LABSE, and found that the
LLM-based evaluator (LLaMA-2-13B) achieves a comparable or higher overall
correlation with human judgments for the considered Indian language pairs.

本研究评估大型语言模型对于自动无参考翻译评估的有效性，并通过模拟人类直接评估的实验来评估英语和印度语言译文的质量。通过构建一个翻译评估任务，我们进行了零样本学习、上下文示例驱动学习和大型语言模型微调，从而提供了一个 0 到 100 的分数，其中 100 表示完美的翻译，1 表示糟糕的翻译。我们将经过训练的系统与现有方法（如 COMET、BERT-Scorer 和 LABSE）进行比较，发现基于大型语言模型的评估器（LLaMA-2-13B）在考虑的印度语言对上实现了相当或更高的整体相关性与人类判断。

面向大型语言模型驱动的无参考翻译评估方法：英语和印度语言

Towards Large Language Model driven Reference-less Translation  Evaluation for English and Indian Languages

Lab results are often confusing and hard to understand. Large language models
(LLMs) such as ChatGPT have opened a promising avenue for patients to get their
questions answered. We aim to assess the feasibility of using LLMs to generate
relevant, accurate, helpful, and unharmful responses to lab test-related
questions asked by patients and to identify potential issues that can be
mitigated with augmentation approaches. We first collected lab test results
related question and answer data from Yahoo! Answers and selected 53 QA pairs
for this study. Using the LangChain framework and ChatGPT web portal, we
generated responses to the 53 questions from four LLMs including GPT-4, Meta
LLaMA 2, MedAlpaca, and ORCA_mini. We first assessed the similarity of their
answers using standard QA similarity-based evaluation metrics including ROUGE,
BLEU, METEOR, BERTScore. We also utilized an LLM-based evaluator to judge
whether a target model has higher quality in terms of relevance, correctness,
helpfulness, and safety than the baseline model. Finally, we performed a manual
evaluation with medical experts for all the responses to seven selected
questions on the same four aspects. The results of Win Rate and medical expert
evaluation both showed that GPT-4's responses achieved better scores than all
the other LLM responses and human responses on all four aspects (relevance,
correctness, helpfulness, and safety). However, LLM responses occasionally also
suffer from a lack of interpretation in one's medical context, incorrect
statements, and lack of references. We find that compared to other three LLMs
and human answer from the Q&A website, GPT-4's responses are more accurate,
helpful, relevant, and safer. However, there are cases which GPT-4 responses
are inaccurate and not individualized. We identified a number of ways to
improve the quality of LLM responses.

利用大型语言模型 (LLMs) 为患者提供对实验室检测相关问题的相关、准确、有帮助和安全的回答的可行性进行评估，并通过采用增强方法来解决潜在问题。我们通过从 Yahoo! Answers 收集实验室测试问题和答案数据，并使用 LangChain 框架和 ChatGPT 网络门户为 53 个问题生成了来自 GPT-4、Meta LLaMA 2、MedAlpaca 和 ORCA_mini 四个 LLM 的回答。通过评估相似度和医学专家的人工评估，结果表明 GPT-4 的回答在相关性、准确性、有帮助性和安全性方面优于其他 LLMs 和人工回答，但偶尔也会出现对医学背景的缺乏解释、错误陈述和缺乏参考等问题。我们发现了提高 LLM 回答质量的多种方法。