The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again

本研究针对少样本学习中的置信区间计算方法存在的问题进行探讨。传统方法基于有替代抽样，导致置信区间的低估。我们提出使用配对检验来部分解决该问题，并探索通过战略性抽样进一步减少置信区间的大小，推动了对少样本学习中置信区间理解的深入。

哎呀，我又抽样了：在少样本学习中重新解读置信区间