Leveraging Large Language Models' remarkable proficiency in text-based tasks,
recent works on Multi-modal LLMs (MLLMs) extend them to other modalities like
vision and audio. However, the progress in these directions has been mostly
focused on tasks that only require a coarse-grained understanding of the
audio-visual semantics. We present Meerkat, an audio-visual LLM equipped with a
fine-grained understanding of image and audio both spatially and temporally.
With a new modality alignment module based on optimal transport and a
cross-attention module that enforces audio-visual consistency, Meerkat can
tackle challenging tasks such as audio referred image grounding, image guided
audio temporal localization, and audio-visual fact-checking. Moreover, we
carefully curate a large dataset AVFIT that comprises 3M instruction tuning
samples collected from open-source datasets, and introduce MeerkatBench that
unifies five challenging audio-visual tasks. We achieve state-of-the-art
performance on all these downstream tasks with a relative improvement of up to
37.12%.

Meerkat 是一个具有对图像和音频进行空间和时间精细理解的音频 - 视觉 LLM 模型，通过优化传输和交叉注意模块，能够处理诸如音频引用图像定位、图像引导音频时间定位和音频 - 视觉事实核查等挑战性任务，在包含 300 万数据样本的 AVFIT 数据集上取得了相对改进高达 37.12% 的最新性能。

Meerkat: 音视大型语言模型在时空中的基线

Meerkat: Audio-Visual Large Language Model for Grounding in Space and  Time

Recording animal behaviour is an important step in evaluating the well-being
of animals and further understanding the natural world. Current methods for
documenting animal behaviour within a zoo setting, such as scan sampling,
require excessive human effort, are unfit for around-the-clock monitoring, and
may produce human-biased results. Several animal datasets already exist that
focus predominantly on wildlife interactions, with some extending to action or
behaviour recognition. However, there is limited data in a zoo setting or data
focusing on the group behaviours of social animals. We introduce a large
meerkat (Suricata Suricatta) behaviour recognition video dataset with diverse
annotated behaviours, including group social interactions, tracking of
individuals within the camera view, skewed class distribution, and varying
illumination conditions. This dataset includes videos from two positions within
the meerkat enclosure at the Wellington Zoo (Wellington, New Zealand), with
848,400 annotated frames across 20 videos and 15 unannotated videos.

引入一个大型、多样化的浣熊行为识别视频数据集，包括群体社交互动、相机视野内个体的跟踪、不同光照条件等，共包括来自新西兰惠灵顿动物园的 20 个视频和 15 个未注释的视频，共计 848,400 个已注释帧。

猫鼬行为识别数据集

Meerkat Behaviour Recognition Dataset

This work proposes to reduce visibility data volume using a
baseline-dependent lossy compression technique that preserves smearing at the
edges of the field-of-view. We exploit the relation of the rank of a matrix and
the fact that a low-rank approximation can describe the raw visibility data as
a sum of basic components where each basic component corresponds to a specific
Fourier component of the sky distribution. As such, the entire visibility data
is represented as a collection of data matrices from baselines, instead of a
single tensor. The proposed methods are formulated as follows: provided a large
dataset of the entire visibility data; the first algorithm, named $simple~SVD$
projects the data into a regular sampling space of rank$-r$ data matrices. In
this space, the data for all the baselines has the same rank, which makes the
compression factor equal across all baselines. The second algorithm, named
$BDSVD$ projects the data into an irregular sampling space of rank$-r_{pq}$
data matrices. The subscript $pq$ indicates that the rank of the data matrix
varies across baselines $pq$, which makes the compression factor
baseline-dependent. MeerKAT and the European Very Long Baseline Interferometry
Network are used as reference telescopes to evaluate and compare the
performance of the proposed methods against traditional methods, such as
traditional averaging and baseline-dependent averaging (BDA). For the same
spatial resolution threshold, both $simple~SVD$ and $BDSVD$ show effective
compression by two-orders of magnitude higher than traditional averaging and
BDA. At the same space-saving rate, there is no decrease in spatial resolution
and there is a reduction in the noise variance in the data which improves the
S/N to over $1.5$ dB at the edges of the field-of-view.

提出了一种基于基线相关的有损压缩技术，可以将能见度数据进行压缩，并且利用矩阵秩的关系以及低秩逼近的方法来描述原始数据，并将其表示为基线数据矩阵的集合，其中每个基本分量分别对应于天空分布的特定傅里叶分量。该方法可以有效压缩数据并且在空间分辨率和噪声降低方面有所改进，其中 MeerKAT 和欧洲长基线干涉仪网络被用于评估和比较所提出的方法。