We present a new method based on episodic Knowledge Graphs (eKGs) for
evaluating (multimodal) conversational agents in open domains. This graph is
generated by interpreting raw signals during conversation and is able to
capture the accumulation of knowledge over time. We apply structural and
semantic analysis of the resulting graphs and translate the properties into
qualitative measures. We compare these measures with existing automatic and
manual evaluation metrics commonly used for conversational agents. Our results
show that our Knowledge-Graph-based evaluation provides more qualitative
insights into interaction and the agent's behavior.

本研究提出了基于知识图谱进行对话代理评估的新方法，通过对话信号生成图谱以表示两者的交互，并将其结构与语义分析转化为定性指标，通过比较自动与人工评估指标来展示其定性分析能力。

通过情节知识图谱评估代理交互

Evaluating Agent Interactions Through Episodic Knowledge Graphs

Citations are generally analyzed using only quantitative measures while
excluding qualitative aspects such as sentiment and intent. However,
qualitative aspects provide deeper insights into the impact of a scientific
research artifact and make it possible to focus on relevant literature free
from bias associated with quantitative aspects. Therefore, it is possible to
rank and categorize papers based on their sentiment and intent. For this
purpose, larger citation sentiment datasets are required. However, from a time
and cost perspective, curating a large citation sentiment dataset is a
challenging task. Particularly, citation sentiment analysis suffers from both
data scarcity and tremendous costs for dataset annotation. To overcome the
bottleneck of data scarcity in the citation analysis domain we explore the
impact of out-domain data during training to enhance the model performance. Our
results emphasize the use of different scheduling methods based on the use
case. We empirically found that a model trained using sequential data
scheduling is more suitable for domain-specific usecases. Conversely, shuffled
data feeding achieves better performance on a cross-domain task. Based on our
findings, we propose an end-to-end trainable multi-task model that covers the
sentiment and intent analysis that utilizes out-domain datasets to overcome the
data scarcity.

通过利用外部数据集，该文提出了一个端到端可训练的多任务模型，覆盖引文情感分析和意图分析，并且可以克服引文分析领域的数据稀缺性问题。

利用外领域数据集增强多任务引文分析

Utilizing Out-Domain Datasets to Enhance Multi-Task Citation Analysis

Generative models, in particular generative adversarial networks (GANs), have
received significant attention recently. A number of GAN variants have been
proposed and have been utilized in many applications. Despite large strides in
terms of theoretical progress, evaluating and comparing GANs remains a daunting
task. While several measures have been introduced, as of yet, there is no
consensus as to which measure best captures strengths and limitations of models
and should be used for fair model comparison. As in other areas of computer
vision and machine learning, it is critical to settle on one or few good
measures to steer the progress in this field. In this paper, I review and
critically discuss more than 24 quantitative and 5 qualitative measures for
evaluating generative models with a particular emphasis on GAN-derived models.
I also provide a set of 7 desiderata followed by an evaluation of whether a
given measure or a family of measures is compatible with them.

本文综述了 24 个量化和 5 个定性评估措施，重点评估基于 GAN 的生成模型，并提供了 7 个期望，并评估了一个给定的措施或措施家族是否与它们兼容。