Optical flow estimation is one of the fundamental tasks in low-level computer
vision, which describes the pixel-wise displacement and can be used in many
other tasks. From the apparent aspect, the optical flow can be viewed as the
correlation between the pixels in consecutive frames, so continuously refining
the correlation volume can achieve an outstanding performance. However, it will
make the method have a catastrophic computational complexity. Not only that,
the error caused by the occlusion regions of the successive frames will be
amplified through the inaccurate warp operation. These challenges can not be
solved only from the apparent view, so this paper rethinks the optical flow
estimation from the kinetics viewpoint.We propose a method combining the
apparent and kinetics information from this motivation. The proposed method
directly predicts the optical flow from the feature extracted from images
instead of building the correlation volume, which will improve the efficiency
of the whole network. Meanwhile, the proposed method involves a new
differentiable warp operation that simultaneously considers the warping and
occlusion. Moreover, the proposed method blends the kinetics feature with the
apparent feature through the novel self-supervised loss function. Furthermore,
comprehensive experiments and ablation studies prove that the proposed novel
insight into how to predict the optical flow can achieve the better performance
of the state-of-the-art methods, and in some metrics, the proposed method
outperforms the correlation-based method, especially in situations containing
occlusion and fast moving. The code will be public.

本文提出了一种新的方法，通过从图像提取特征直接预测光流，同时考虑到光流估计的运动学视角和外观视角，采用可微分的变形操作解决了由于运动不准确造成的误差及遮挡问题，并通过自监督损失函数将运动特征与外观特征相结合，实验证明该方法在光流预测上表现优于现有方法，尤其在包含遮挡和快速运动的情况下效果更好。

重新思考以动力学视角预测光流

Rethink Predicting the Optical Flow with the Kinetics Perspective

Cross-modal recipe retrieval has recently gained substantial attention due to
the importance of food in people's lives, as well as the availability of vast
amounts of digital cooking recipes and food images to train machine learning
models. In this work, we revisit existing approaches for cross-modal recipe
retrieval and propose a simplified end-to-end model based on well established
and high performing encoders for text and images. We introduce a hierarchical
recipe Transformer which attentively encodes individual recipe components
(titles, ingredients and instructions). Further, we propose a self-supervised
loss function computed on top of pairs of individual recipe components, which
is able to leverage semantic relationships within recipes, and enables training
using both image-recipe and recipe-only samples. We conduct a thorough analysis
and ablation studies to validate our design choices. As a result, our proposed
method achieves state-of-the-art performance in the cross-modal recipe
retrieval task on the Recipe1M dataset. We make code and models publicly
available.

本研究针对跨模态食谱检索任务，提出一个基于端到端模型的层级食谱 Transformer，使用自我监督损失函数在图像和食谱样本上训练，取得了 Recipe1M 数据集上的最佳性能表现。

层次变换器与自监督学习优化跨模态菜谱检索

Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers  and Self-supervised Learning

Objects moving at high speed appear significantly blurred when captured with
cameras. The blurry appearance is especially ambiguous when the object has
complex shape or texture. In such cases, classical methods, or even humans, are
unable to recover the object's appearance and motion. We propose a method that,
given a single image with its estimated background, outputs the object's
appearance and position in a series of sub-frames as if captured by a
high-speed camera (i.e. temporal super-resolution). The proposed generative
model embeds an image of the blurred object into a latent space representation,
disentangles the background, and renders the sharp appearance. Inspired by the
image formation model, we design novel self-supervised loss function terms that
boost performance and show good generalization capabilities. The proposed DeFMO
method is trained on a complex synthetic dataset, yet it performs well on
real-world data from several datasets. DeFMO outperforms the state of the art
and generates high-quality temporal super-resolution frames.

该研究提出了一种利用生成模型和自监督损失函数训练的 DeFMO 方法，在仅有单张图像和背景估计的情况下，以高速相机的子帧形式输出物体在各个位置的外观和运动状态，实现了 temporal super-resolution，效果优于现有技术并适用于真实数据。