Fusion of a panchromatic (PAN) image and corresponding multispectral (MS)
image is also known as pansharpening, which aims to combine abundant spatial
details of PAN and spectral information of MS. Due to the absence of
high-resolution MS images, available deep-learning-based methods usually follow
the paradigm of training at reduced resolution and testing at both reduced and
full resolution. When taking original MS and PAN images as inputs, they always
obtain sub-optimal results due to the scale variation. In this paper, we
propose to explore the self-supervised representation of pansharpening by
designing a cross-predictive diffusion model, named CrossDiff. It has two-stage
training. In the first stage, we introduce a cross-predictive pretext task to
pre-train the UNet structure based on conditional DDPM, while in the second
stage, the encoders of the UNets are frozen to directly extract spatial and
spectral features from PAN and MS, and only the fusion head is trained to adapt
for pansharpening task. Extensive experiments show the effectiveness and
superiority of the proposed model compared with state-of-the-art supervised and
unsupervised methods. Besides, the cross-sensor experiments also verify the
generalization ability of proposed self-supervised representation learners for
other satellite's datasets. We will release our code for reproducibility.

提出了一种名为 CrossDiff 的跨预测扩散模型，利用自我监督的表示学习方法进行卫星图像的泛光锐化，通过两阶段的训练并与其他方法比较，实验证明了该模型的有效性和优越性，同时验证了其在其他卫星数据集上的泛化能力。

CrossDiff: 基于交叉预测扩散模型探索自监督全色影像增强的表示

CrossDiff: Exploring Self-Supervised Representation of Pansharpening via  Cross-Predictive Diffusion Model

Decoding visual stimuli from brain recordings aims to deepen our
understanding of the human visual system and build a solid foundation for
bridging human and computer vision through the Brain-Computer Interface.
However, reconstructing high-quality images with correct semantics from brain
recordings is a challenging problem due to the complex underlying
representations of brain signals and the scarcity of data annotations. In this
work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned
Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an
effective self-supervised representation of fMRI data using mask modeling in a
large latent space inspired by the sparse coding of information in the primary
visual cortex. Then by augmenting a latent diffusion model with
double-conditioning, we show that MinD-Vis can reconstruct highly plausible
images with semantically matching details from brain recordings using very few
paired annotations. We benchmarked our model qualitatively and quantitatively;
the experimental results indicate that our method outperformed state-of-the-art
in both semantic mapping (100-way semantic classification) and generation
quality (FID) by 66% and 41% respectively. An exhaustive ablation study was
also conducted to analyze our framework.

本文提出了一种使用双条件潜在扩散模型的 MinD-Vis：稀疏掩蔽脑建模方法，通过大型潜在空间中的掩蔽建模来学习 fMRI 数据的有效自我监督表示。实验证明，该方法在语义映射和生成质量方面都优于现有技术，并且只需很少的成对注释即可从脑记录中重建高度合理的图像。

超越大脑：使用稀疏掩码建模的条件扩散模型进行视觉解码

Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked  Modeling for Vision Decoding

Unsupervised speech representations have taken off, with benchmarks (SUPERB,
ZeroSpeech) demonstrating major progress on semi-supervised speech recognition,
speech synthesis, and speech-only language modelling. Inspiration comes from
the promise of ``discovering the phonemes'' of a language or a similar
low-bitrate encoding. However, one of the critical properties of phoneme
transcriptions is context-invariance: the phonetic context of a speech sound
can have massive influence on the way it is pronounced, while the text remains
stable. This is what allows tokens of the same word to have the same
transcriptions -- key to language understanding. Current benchmarks do not
measure context-invariance. We develop a new version of the ZeroSpeech ABX
benchmark that measures context-invariance, and apply it to recent
self-supervised representations. We demonstrate that the context-independence
of representations is predictive of the stability of word-level
representations. We suggest research concentrate on improving
context-independence of self-supervised and unsupervised representations.

本文介绍了语音或文本的不变性对于理解语言至关重要，提出了一种新的 ZeroSpeech ABX 基准测试方法来测试表示的上下文独立性，并建议将重点放在改善自监督和无监督表示的上下文独立性上。