User interface (UI) design is a difficult yet important task for ensuring the
usability, accessibility, and aesthetic qualities of applications. In our
paper, we develop a machine-learned model, UIClip, for assessing the design
quality and visual relevance of a UI given its screenshot and natural language
description. To train UIClip, we used a combination of automated crawling,
synthetic augmentation, and human ratings to construct a large-scale dataset of
UIs, collated by description and ranked by design quality. Through training on
the dataset, UIClip implicitly learns properties of good and bad designs by i)
assigning a numerical score that represents a UI design's relevance and quality
and ii) providing design suggestions. In an evaluation that compared the
outputs of UIClip and other baselines to UIs rated by 12 human designers, we
found that UIClip achieved the highest agreement with ground-truth rankings.
Finally, we present three example applications that demonstrate how UIClip can
facilitate downstream applications that rely on instantaneous assessment of UI
design quality: i) UI code generation, ii) UI design tips generation, and iii)
quality-aware UI example search.

通过机器学习模型 UIClip，本研究开发了一种用于评估 UI 设计质量和视觉相关性的方法，通过截图和自然语言描述来对 UI 进行评估，实验证明 UIClip 在与其他基准方法和人工设计师评分进行对比时取得了最高一致性，从而为 UI 设计质量的直接评估提供了便利。

UIClip：一个基于数据的用户界面设计评估模型

UIClip: A Data-driven Model for Assessing User Interface Design

Video search has become the main routine for users to discover videos
relevant to a text query on large short-video sharing platforms. During
training a query-video bi-encoder model using online search logs, we identify a
modality bias phenomenon that the video encoder almost entirely relies on text
matching, neglecting other modalities of the videos such as vision, audio. This
modality imbalanceresults from a) modality gap: the relevance between a query
and a video text is much easier to learn as the query is also a piece of text,
with the same modality as the video text; b) data bias: most training samples
can be solved solely by text matching. Here we share our practices to improve
the first retrieval stage including our solution for the modality imbalance
issue. We propose MBVR (short for Modality Balanced Video Retrieval) with two
key components: manually generated modality-shuffled (MS) samples and a dynamic
margin (DM) based on visual relevance. They can encourage the video encoder to
pay balanced attentions to each modality. Through extensive experiments on a
real world dataset, we show empirically that our method is both effective and
efficient in solving modality bias problem. We have also deployed our MBVR in a
large video platform and observed statistically significant boost over a highly
optimized baseline in an A/B test and manual GSB evaluations.

介绍了将 MBVR 应用于大型视频平台的实践，该方法采用人工生成的模态混洗样本和基于视觉相关性的动态边距（DM），旨在鼓励视频编码器对每种模态进行平衡关注，通过实验证明了该方法在解决模态偏见问题方面的有效性和高效性。

视频检索的模态平衡嵌入

Modality-Balanced Embedding for Video Retrieval

The contemporary visual captioning models frequently hallucinate objects that
are not actually in a scene, due to the visual misclassification or
over-reliance on priors that resulting in the semantic inconsistency between
the visual information and the target lexical words. The most common way is to
encourage the captioning model to dynamically link generated object words or
phrases to appropriate regions of the image, i.e., the grounded image
captioning (GIC). However, GIC utilizes an auxiliary task (grounding objects)
that has not solved the key issue of object hallucination, i.e., the semantic
inconsistency. In this paper, we take a novel perspective on the issue above -
exploiting the semantic coherency between the visual and language modalities.
Specifically, we propose the Consensus Rraph Representation Learning framework
(CGRL) for GIC that incorporates a consensus representation into the grounded
captioning pipeline. The consensus is learned by aligning the visual graph
(e.g., scene graph) to the language graph that consider both the nodes and
edges in a graph. With the aligned consensus, the captioning model can capture
both the correct linguistic characteristics and visual relevance, and then
grounding appropriate image regions further. We validate the effectiveness of
our model, with a significant decline in object hallucination (-9% CHAIRi) on
the Flickr30k Entities dataset. Besides, our CGRL also evaluated by several
automatic metrics and human evaluation, the results indicate that the proposed
approach can simultaneously improve the performance of image captioning (+2.9
Cider) and grounding (+2.3 F1LOC).

本文针对基于场景图的图像描述模型中存在的物体虚幻问题，提出了利用语义完整性和视觉相关性混合的 CGRL 框架，以提高图像描述和定位准确性。结果表明，该方法显著降低了物体虚幻的发生率，同时提高了图像描述和物体定位的质量。