Neural networks are trained by minimizing a loss function that defines the discrepancy between the predicted model output and the target value. The selection of the loss function is crucial to achieve task-specific behaviour and highly influences the capability of the model. A variety of loss functions have been proposed for a wide range of tasks affecting training and model performance. For classification tasks, the cross entropy is the de-facto standard and usually the first choice. Here, we try to experimentally challenge the well-known loss functions, including cross entropy (CE) loss, by utilizing the genetic programming (GP) approach, a population-based evolutionary algorithm. GP constructs loss functions from a set of operators and leaf nodes and these functions are repeatedly recombined and mutated to find an optimal structure. Experiments were carried out on different small-sized datasets CIFAR-10, CIFAR-100 and Fashion-MNIST using an Inception model. The 5 best functions found were evaluated for different model architectures on a set of standard datasets ranging from 2 to 102 classes and very different sizes. One function, denoted as Next Generation Loss (NGL), clearly stood out showing same or better performance for all tested datasets compared to CE. To evaluate the NGL function on a large-scale dataset, we tested its performance on the Imagenet-1k dataset where it showed improved top-1 accuracy compared to models trained with identical settings and other losses. Finally, the NGL was trained on a segmentation downstream task for Pascal VOC 2012 and COCO-Stuff164k datasets improving the underlying model performance.

神经网络通过最小化损失函数来学习，定义了预测模型输出与目标值之间的差异。选择损失函数对于实现特定任务行为至关重要，并且极大地影响模型的能力。我们利用遗传编程方法对众所周知的损失函数进行了实验性挑战，包括交叉熵损失，提出了5个最佳函数，并在各种模型架构的标准数据集上进行了评估，结果发现Next Generation Loss（NGL）函数在所有测试数据集上表现出相同或更好的性能，包括Imagetnet-1k数据集。最后，我们将NGL函数用于Pascal VOC 2012和COCO-Stuff164k数据集的分割后任务训练，提高了底层模型的性能。

图像分类的下一代损失函数