Public benchmarks play an essential role in the evaluation of large language
models. However, data contamination can lead to inflated performance, rendering
them unreliable for model comparison. It is therefore crucial to detect
contamination and estimate its impact on measured performance. Unfortunately,
existing detection methods can be easily evaded and fail to quantify
contamination. To overcome these limitations, we propose a novel definition of
contamination as artificially inflated and non-generalizing benchmark
performance instead of the inclusion of benchmark samples in the training data.
This perspective enables us to detect any model with inflated performance,
i.e., performance that does not generalize to rephrased samples, synthetic
samples from the same distribution, or different benchmarks for the same task.
Based on this insight, we develop ConStat, a statistical method that reliably
detects and quantifies contamination by comparing performance between a primary
and reference benchmark relative to a set of reference models. We demonstrate
the effectiveness of ConStat in an extensive evaluation of diverse model
architectures, benchmarks, and contamination scenarios and find high levels of
contamination in multiple popular models including Mistral, Llama, Yi, and the
top-3 Open LLM Leaderboard models.

公共基准测试、数据污染、性能、检测方法、ConStat

ConStat：大型语言模型中基于性能的污染检测

ConStat: Performance-Based Contamination Detection in Large Language  Models

Semi-supervised semantic segmentation involves assigning pixel-wise labels to
unlabeled images at training time. This is useful in a wide range of real-world
applications where collecting pixel-wise labels is not feasible in time or
cost. Current approaches to semi-supervised semantic segmentation work by
predicting pseudo-labels for each pixel from a class-wise probability
distribution output by a model. If the predicted probability distribution is
incorrect, however, this leads to poor segmentation results, which can have
knock-on consequences in safety critical systems, like medical images or
self-driving cars. It is, therefore, important to understand what a model does
not know, which is mainly achieved by uncertainty quantification. Recently,
neural processes (NPs) have been explored in semi-supervised image
classification, and they have been a computationally efficient and effective
method for uncertainty quantification. In this work, we move one step forward
by adapting NPs to semi-supervised semantic segmentation, resulting in a new
model called NP-SemiSeg. We experimentally evaluated NP-SemiSeg on the public
benchmarks PASCAL VOC 2012 and Cityscapes, with different training settings,
and the results verify its effectiveness.

通过将神经过程（NPs）应用于半监督语义分割，我们提出了一种新的模型 NP-SemiSeg，并在公共基准数据集 PASCAL VOC 2012 和 Cityscapes 上进行了实验验证其有效性。

NP-SemiSeg：当神经过程遇上半监督语义分割

NP-SemiSeg: When Neural Processes meet Semi-Supervised Semantic  Segmentation

This paper proposes a novel multi-modal transformer network for detecting
actions in untrimmed videos. To enrich the action features, our transformer
network utilizes a new multi-modal attention mechanism that computes the
correlations between different spatial and motion modalities combinations.
Exploring such correlations for actions has not been attempted previously. To
use the motion and spatial modality more effectively, we suggest an algorithm
that corrects the motion distortion caused by camera movement. Such motion
distortion, common in untrimmed videos, severely reduces the expressive power
of motion features such as optical flow fields. Our proposed algorithm
outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and
ActivityNet. We also conducted comparative experiments on our new instructional
activity dataset, including a large set of challenging classroom videos
captured from elementary schools.

本文提出了一种新的多模态转换器网络，用于检测未修剪视频中的动作，利用新的多模态注意机制计算不同空间和动态模态组合之间的相关性，并提出一种算法来纠正相机运动引起的动态变形，该算法在 THUMOS14 和 ActivityNet 两个公共基准测试上优于现有方法，在我们的新教育活动数据集上进行了比较实验，该数据集包括从小学捕捉的大量具有挑战性的课堂视频。

多模态 Transformer 网络用于动作检测

A Multi-Modal Transformer Network for Action Detection

We address the weakly supervised video highlight detection problem for
learning to detect segments that are more attractive in training videos given
their video event label but without expensive supervision of manually
annotating highlight segments. While manually averting localizing highlight
segments, weakly supervised modeling is challenging, as a video in our daily
life could contain highlight segments with multiple event types, e.g., skiing
and surfing. In this work, we propose casting weakly supervised video highlight
detection modeling for a given specific event as a multiple instance ranking
network (MINI-Net) learning. We consider each video as a bag of segments, and
therefore, the proposed MINI-Net learns to enforce a higher highlight score for
a positive bag that contains highlight segments of a specific event than those
for negative bags that are irrelevant. In particular, we form a max-max ranking
loss to acquire a reliable relative comparison between the most likely positive
segment instance and the hardest negative segment instance. With this max-max
ranking loss, our MINI-Net effectively leverages all segment information to
acquire a more distinct video feature representation for localizing the
highlight segments of a specific event in a video. The extensive experimental
results on three challenging public benchmarks clearly validate the efficacy of
our multiple instance ranking approach for solving the problem.

本文提出了一种弱监督视频亮点检测建模方法，将其作为多实例排名网络（MINI-Net）学习，并使用 max-max 排名损失以获取可靠的相对比较，实现在视频中定位特定事件的高亮片段，实验证明其在三大公共基准测试中的有效性。