Addressing the challenge of a digital assistant capable of executing a wide
array of user tasks, our research focuses on the realm of instruction-based
mobile device control. We leverage recent advancements in large language models
(LLMs) and present a visual language model (VLM) that can fulfill diverse tasks
on mobile devices. Our model functions by interacting solely with the user
interface (UI). It uses the visual input from the device screen and mimics
human-like interactions, encompassing gestures such as tapping and swiping.
This generality in the input and output space allows our agent to interact with
any application on the device. Unlike previous methods, our model operates not
only on a single screen image but on vision-language sentences created from
sequences of past screenshots along with corresponding actions. Evaluating our
method on the challenging Android in the Wild benchmark demonstrates its
promising efficacy and potential.

利用大型语言模型和视觉语言模型，我们的研究致力于解决数字助理执行各种用户任务的挑战，特别是在基于指令的移动设备控制领域。通过与用户界面进行交互，我们的模型利用设备屏幕的视觉输入并模拟人类般的交互，包括点击和滑动等手势。这种输入和输出空间的广泛适用性使得我们的代理能够与设备上的任何应用程序进行交互。与以往方法不同的是，我们的模型不仅仅操作单个屏幕图像，还通过过去截图序列和相应操作生成视觉语言句子。在具有挑战性的 “Android in the Wild” 基准测试中评估我们的方法表明其有效性和潜力。

训练一种视觉语言模型作为智能手机助手

Training a Vision Language Model as Smartphone Assistant

Conventional approaches to facial expression recognition primarily focus on
the classification of six basic facial expressions. Nevertheless, real-world
situations present a wider range of complex compound expressions that consist
of combinations of these basics ones due to limited availability of
comprehensive training datasets. The 6th Workshop and Competition on Affective
Behavior Analysis in-the-wild (ABAW) offered unlabeled datasets containing
compound expressions. In this study, we propose a zero-shot approach for
recognizing compound expressions by leveraging a pretrained visual language
model integrated with some traditional CNN networks.

利用预训练的视觉语言模型结合传统的 CNN 网络，提出了一种用于识别合成表情的零样本方法。

第六届 ABAW 挑战赛上基于视觉语言模型的零样本复合表达识别

Zero-shot Compound Expression Recognition with Visual Language Model at  the 6th ABAW Challenge

We introduce PhotoBot, a framework for automated photo acquisition based on
an interplay between high-level human language guidance and a robot
photographer. We propose to communicate photography suggestions to the user via
a reference picture that is retrieved from a curated gallery. We exploit a
visual language model (VLM) and an object detector to characterize reference
pictures via textual descriptions and use a large language model (LLM) to
retrieve relevant reference pictures based on a user's language query through
text-based reasoning. To correspond the reference picture and the observed
scene, we exploit pre-trained features from a vision transformer capable of
capturing semantic similarity across significantly varying images. Using these
features, we compute pose adjustments for an RGB-D camera by solving a
Perspective-n-Point (PnP) problem. We demonstrate our approach on a real-world
manipulator equipped with a wrist camera. Our user studies show that photos
taken by PhotoBot are often more aesthetically pleasing than those taken by
users themselves, as measured by human feedback.

PhotoBot 框架结合高级人类语言引导和机器摄影师，实现了基于自动话术采集照片的功能。利用视觉语言模型（VLM）和目标检测器，通过文本描述表征参考图片，然后通过大型语言模型（LLM）根据用户的语言查询检索相关的参考图片。利用一个能够跨不同图片捕捉语义相似性的视觉转换器的训练特征，来对应参考图片和观察场景。该方法通过求解透视 n 点（PnP）问题计算了 RGB-D 相机的姿势调整。在配备腕部相机的真实世界操作器上演示了我们的方法。我们的用户研究表明，通过 PhotoBot 拍摄的照片在美学上往往比用户自己拍摄的更令人满意，这可通过人类反馈来衡量。

PhotoBot：基于自然语言引导的交互式摄影

PhotoBot: Reference-Guided Interactive Photography via Natural Language

The goal of vision-language modeling is to allow models to tie language
understanding with visual inputs. The aim of this paper is to evaluate and
align the Visual Language Model (VLM) called Multimodal Augmentation of
Generative Models through Adapter-based finetuning (MAGMA) with human values.
MAGMA is a VLM that is capable of image captioning and visual
question-answering. We will evaluate its alignment in three different
scenarios. To begin, we assess MAGMA's out-of-the-box alignment through the
checkpoint provided by Hugging Face. Then, we measure if few-shot learning
manages to improve the results. Finally, we finetune the model on aligned
examples and evaluate its behavior.

本文旨在评估并对齐名为 Multimodal Augmentation of Generative Models through Adapter-based finetuning (MAGMA) 的视觉语言模型与人的价值观，通过三个不同的情景来评估其对齐情况，并测试其在图像字幕和视觉问答方面的能力。