Pre-trained models have been a foundational approach in speech recognition,
albeit with associated additional costs. In this study, we propose a
regularization technique that facilitates the training of visual and
audio-visual speech recognition models (VSR and AVSR) from scratch. This
approach, abbreviated as \textbf{MSRS} (Multimodal Speech Recognition from
Scratch), introduces a sparse regularization that rapidly learns sparse
structures within the dense model at the very beginning of training, which
receives healthier gradient flow than the dense equivalent. Once the sparse
mask stabilizes, our method allows transitioning to a dense model or keeping a
sparse model by updating non-zero values. MSRS achieves competitive results in
VSR and AVSR with 21.1% and 0.9% WER on the LRS3 benchmark, while reducing
training time by at least 2x. We explore other sparse approaches and show that
only MSRS enables training from scratch by implicitly masking the weights
affected by vanishing gradients.

该研究提出了一种正则化技术，可以从头开始训练视觉和视听语音识别模型，通过学习稀疏结构并减少训练时间，同时达到竞争性的识别结果。

MSRS: 用稀疏掩码优化从零开始训练多模态语音识别模型

MSRS: Training Multimodal Speech Recognition Models from Scratch with  Sparse Mask Optimization

Label smoothing is a widely used technique in various domains, such as image
classification and speech recognition, known for effectively combating model
overfitting. However, there is few research on its application to text
sentiment classification. To fill in the gap, this study investigates the
implementation of label smoothing for sentiment classification by utilizing
different levels of smoothing. The primary objective is to enhance sentiment
classification accuracy by transforming discrete labels into smoothed label
distributions. Through extensive experiments, we demonstrate the superior
performance of label smoothing in text sentiment classification tasks across
eight diverse datasets and deep learning architectures: TextCNN, BERT, and
RoBERTa, under two learning schemes: training from scratch and fine-tuning.

这项研究通过使用不同程度的平滑化技术来实现情感分类的标签平滑化，从而提高情感分类准确性，并在八个不同的数据集和深度学习架构（TextCNN、BERT 和 RoBERTa）中的训练和微调两种学习方案下，通过大量实验展示了标签平滑化在文本情感分类任务中的卓越性能。

文本情感分类的标签平滑优化

Label Smoothing for Enhanced Text Sentiment Classification

Few-shot named entity recognition (NER) aims at identifying named entities
based on only few labeled instances. Current few-shot NER methods focus on
leveraging existing datasets in the rich-resource domains which might fail in a
training-from-scratch setting where no source-domain data is used. To tackle
training-from-scratch setting, it is crucial to make full use of the annotation
information (the boundaries and entity types). Therefore, in this paper, we
propose a novel multi-task (Seed, Expand and Entail) learning framework,
SEE-Few, for Few-shot NER without using source domain data. The seeding and
expanding modules are responsible for providing as accurate candidate spans as
possible for the entailing module. The entailing module reformulates span
classification as a textual entailment task, leveraging both the contextual
clues and entity type information. All the three modules share the same text
encoder and are jointly learned. Experimental results on four benchmark
datasets under the training-from-scratch setting show that the proposed method
outperformed state-of-the-art few-shot NER methods with a large margin. Our
code is available at this https URL.

本文提出了一种新颖的多任务学习框架 SEE-Few，用于在没有使用源域数据的情况下进行少样本实体识别，该框架通过种子、扩展和蕴含的学习模块来提高实体识别的准确性，实验结果表明，该方法在四个基准数据集上表现出了比现有技术高出很大的优势。

SEE-Few: 小样本命名实体识别的种子、扩张与蕴含

SEE-Few: Seed, Expand and Entail for Few-shot Named Entity Recognition

We propose Deeply Supervised Object Detectors (DSOD), an object detection
framework that can be trained from scratch. Recent advances in object detection
heavily depend on the off-the-shelf models pre-trained on large-scale
classification datasets like ImageNet and OpenImage. However, one problem is
that adopting pre-trained models from classification to detection task may
incur learning bias due to the different objective function and diverse
distributions of object categories. Techniques like fine-tuning on detection
task could alleviate this issue to some extent but are still not fundamental.
Furthermore, transferring these pre-trained models across discrepant domains
will be more difficult (e.g., from RGB to depth images). Thus, a better
solution to handle these critical problems is to train object detectors from
scratch, which motivates our proposed method. Previous efforts on this
direction mainly failed by reasons of the limited training data and naive
backbone network structures for object detection. In DSOD, we contribute a set
of design principles for learning object detectors from scratch. One of the key
principles is the deep supervision, enabled by layer-wise dense connections in
both backbone networks and prediction layers, plays a critical role in learning
good detectors from scratch. After involving several other principles, we build
our DSOD based on the single-shot detection framework (SSD). We evaluate our
method on PASCAL VOC 2007, 2012 and COCO datasets. DSOD achieves consistently
better results than the state-of-the-art methods with much more compact models.
Specifically, DSOD outperforms baseline method SSD on all three benchmarks,
while requiring only 1/2 parameters. We also observe that DSOD can achieve
comparable/slightly better results than Mask RCNN + FPN (under similar input
size) with only 1/3 parameters, using no extra data or pre-trained models.

提出 Deeply Supervised Object Detectors（DSOD）框架，实现可以从头开始训练的目标检测器。通过 Layer-wise Dense 连接在主干网和预测层之间实现深度监督，DSOD 能在 PASCAL VOC 和 COCO 数据集上取得比现有方法更好的检测结果，并使用更小的模型参数。