Learning meaningful representations of complex objects that can be seen
through multiple ($k\geq 3$) views or modalities is a core task in machine
learning. Existing methods use losses originally intended for paired views, and
extend them to $k$ views, either by instantiating $\tfrac12k(k-1)$ loss-pairs,
or by using reduced embeddings, following a \textit{one vs. average-of-rest}
strategy. We propose the multi-marginal matching gap (M3G), a loss that borrows
tools from multi-marginal optimal transport (MM-OT) theory to simultaneously
incorporate all $k$ views. Given a batch of $n$ points, each seen as a
$k$-tuple of views subsequently transformed into $k$ embeddings, our loss
contrasts the cost of matching these $n$ ground-truth $k$-tuples with the MM-OT
polymatching cost, which seeks $n$ optimally arranged $k$-tuples chosen within
these $n\times k$ vectors. While the exponential complexity $O(n^k$) of the
MM-OT problem may seem daunting, we show in experiments that a suitable
generalization of the Sinkhorn algorithm for that problem can scale to, e.g.,
$k=3\sim 6$ views using mini-batches of size $64~\sim128$. Our experiments
demonstrate improved performance over multiview extensions of pairwise losses,
for both self-supervised and multimodal tasks.

复杂对象的多视图学习中，多极值匹配缺口 (M3G) 是一种损失函数，它从多极值最优输运 (MM-OT) 理论中借用工具，同时结合所有视图，并在自我监督和多模态任务中展示了比成对损失更好的性能。

用多边匹配差异对比多重表示

Contrasting Multiple Representations with the Multi-Marginal Matching  Gap

Text-to-image generation models have seen considerable advancement, catering
to the increasing interest in personalized image creation. Current
customization techniques often necessitate users to provide multiple images
(typically 3-5) for each customized object, along with the classification of
these objects and descriptive textual prompts for scenes. This paper questions
whether the process can be made more user-friendly and the customization more
intricate. We propose a method where users need only provide images along with
text for each customization topic, and necessitates only a single image per
visual concept. We introduce the concept of a ``multi-modal prompt'', a novel
integration of text and images tailored to each customization concept, which
simplifies user interaction and facilitates precise customization of both
objects and scenes. Our proposed paradigm for customized text-to-image
generation surpasses existing finetune-based methods in user-friendliness and
the ability to customize complex objects with user-friendly inputs. Our code is
available at
$\href{https://github.com/zhongzero/Multi-Modal-Prompt}{this https URL}$.

本文提出了一种方法，用户只需提供每个定制主题的图像和文本，而不需要提供多张图像，通过引入 “多模态提示” 概念，将文本和图像相结合，简化用户交互，实现对对象和场景的精确定制。该方法在用户友好性和使用友好输入定制复杂对象方面超过了现有的基于微调方法。

多模式提示下的用户友好定制生成

User-Friendly Customized Generation with Multi-Modal Prompts

The goal of our work is to generate high-quality novel views from monocular
videos of complex and dynamic scenes. Prior methods, such as DynamicNeRF, have
shown impressive performance by leveraging time-varying dynamic radiation
fields. However, these methods have limitations when it comes to accurately
modeling the motion of complex objects, which can lead to inaccurate and blurry
renderings of details. To address this limitation, we propose a novel approach
that builds upon a recent generalization NeRF, which aggregates nearby views
onto new viewpoints. However, such methods are typically only effective for
static scenes. To overcome this challenge, we introduce a module that operates
in both the time and frequency domains to aggregate the features of object
motion. This allows us to learn the relationship between frames and generate
higher-quality images. Our experiments demonstrate significant improvements
over state-of-the-art methods on dynamic scene datasets. Specifically, our
approach outperforms existing methods in terms of both the accuracy and visual
quality of the synthesized views.

我们的研究目标是从复杂动态场景的单目视频中生成高质量的新视图。为了解决过去方法在准确建模复杂物体运动方面的局限性，我们提出了一种新方法，结合了近期的广义 NeRF 和在时间和频率域中操作的模块，以改善动态场景的图像合成的精确度和视觉质量。