AI services are known to have unstable behavior when subjected to changes in
data, models or users. Such behaviors, whether triggered by omission or
commission, lead to trust issues when AI works with humans. The current
approach of assessing AI services in a black box setting, where the consumer
does not have access to the AI's source code or training data, is limited. The
consumer has to rely on the AI developer's documentation and trust that the
system has been built as stated. Further, if the AI consumer reuses the service
to build other services which they sell to their customers, the consumer is at
the risk of the service providers (both data and model providers). Our
approach, in this context, is inspired by the success of nutritional labeling
in food industry to promote health and seeks to assess and rate AI services for
trust from the perspective of an independent stakeholder. The ratings become a
means to communicate the behavior of AI systems so that the consumer is
informed about the risks and can make an informed decision. In this paper, we
will first describe recent progress in developing rating methods for text-based
machine translator AI services that have been found promising with user
studies. Then, we will outline challenges and vision for a principled,
multi-modal, causality-based rating methodologies and its implication for
decision-support in real-world scenarios like health and food recommendation.

基于食品行业中营养标签的成功，本文旨在评估和评级 AI 服务的用户信任度，并提出了面向现实世界场景如健康和食品建议的原则性、多模式、因果关系评级方法的挑战和远景。

自动化评估文本处理服务的可信度的进展

Advances in Automatically Rating the Trustworthiness of Text Processing Services

To provide users insight into the value and limits of world university
rankings, a comparative analysis is conducted of 5 ranking systems: ARWU,
Leiden, THE, QS and U-Multirank. It links these systems with one another at the
level of individual institutions, and analyses the overlap in institutional
coverage, geographical coverage, how indicators are calculated from raw data,
the skewness of indicator distributions, and statistical correlations between
indicators. Four secondary analyses are presented investigating national
academic systems and selected pairs of indicators. It is argued that current
systems are still one-dimensional in the sense that they provide finalized,
seemingly unrelated indicator values rather than offering a data set and tools
to observe patterns in multi-faceted data. By systematically comparing
different systems, more insight is provided into how their institutional
coverage, rating methods, the selection of indicators and their normalizations
influence the ranking positions of given institutions.

本文就全球大学排名的价值和局限性进行了比较分析，包括 ARWU, Leiden, THE, QS 和 U-Multirank 等 5 个排名系统，并分析了它们之间的覆盖率、计算指标的方法、指标分布的偏斜以及指标之间的统计相关性。通过对这些排名系统的对比，提供了更多关于它们的机构覆盖范围、评分方法、指标选择和规范化如何影响给定机构排名和定位的见解。