Multilingual pre-trained contextual embedding models (Devlin et al., 2019)
have achieved impressive performance on zero-shot cross-lingual transfer tasks.
Finding the most effective fine-tuning strategy to fine-tune these models on
high-resource languages so that it transfers well to the zero-shot languages is
a non-trivial task. In this paper, we propose a novel meta-optimizer to
soft-select which layers of the pre-trained model to freeze during fine-tuning.
We train the meta-optimizer by simulating the zero-shot transfer scenario.
Results on cross-lingual natural language inference show that our approach
improves over the simple fine-tuning baseline and X-MAML (Nooralahzadeh et al.,
2020).

该论文提出了一种新的元优化器方法，用于在高资源语言上对多语言预训练上下文嵌入模型进行微调以实现零样本跨语言转移，该方法优于基线模型和 X-MAML 模型。

基于元学习的零样本跨语言软层选择

Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual  Transfer

Learning to optimize has emerged as a powerful framework for various
optimization and machine learning tasks. Current such "meta-optimizers" often
learn in the space of continuous optimization algorithms that are point-based
and uncertainty-unaware. To overcome the limitations, we propose a
meta-optimizer that learns in the algorithmic space of both point-based and
population-based optimization algorithms. The meta-optimizer targets at a
meta-loss function consisting of both cumulative regret and entropy.
Specifically, we learn and interpret the update formula through a population of
LSTMs embedded with sample- and feature-level attentions. Meanwhile, we
estimate the posterior directly over the global optimum and use an uncertainty
measure to help guide the learning process. Empirical results over non-convex
test functions and the protein-docking application demonstrate that this new
meta-optimizer outperforms existing competitors.

本文提出了一种新的元优化器，该优化器能够在基于点和基于种群的优化算法算法空间中进行学习，提高了预测的准确性和优化能力，并在非凸测试函数和蛋白质对接应用中的实验结果表明，该元优化器优于现有竞争对手。