Physics-informed neural networks (PINNs) are known to suffer from
optimization difficulty. In this work, we reveal the connection between the
optimization difficulty of PINNs and activation functions. Specifically, we
show that PINNs exhibit high sensitivity to activation functions when solving
PDEs with distinct properties. Existing works usually choose activation
functions by inefficient trial-and-error. To avoid the inefficient manual
selection and to alleviate the optimization difficulty of PINNs, we introduce
adaptive activation functions to search for the optimal function when solving
different problems. We compare different adaptive activation functions and
discuss their limitations in the context of PINNs. Furthermore, we propose to
tailor the idea of learning combinations of candidate activation functions to
the PINNs optimization, which has a higher requirement for the smoothness and
diversity on learned functions. This is achieved by removing activation
functions which cannot provide higher-order derivatives from the candidate set
and incorporating elementary functions with different properties according to
our prior knowledge about the PDE at hand. We further enhance the search space
with adaptive slopes. The proposed adaptive activation function can be used to
solve different PDE systems in an interpretable way. Its effectiveness is
demonstrated on a series of benchmarks. Code is available at
this https URL

通过引入自适应激活函数来解决物理信息神经网络 (PINNs) 的优化困难，该方法可以有效地解决不同问题中的偏微分方程组，并在一系列基准测试中展现了其有效性。

学习为基于物理知识的神经网络设计特定的激活函数

Learning Specialized Activation Functions for Physics-informed Neural  Networks

Quantized neural networks with low-bit weights and activations are attractive
for developing AI accelerators. However, the quantization functions used in
most conventional quantization methods are non-differentiable, which increases
the optimization difficulty of quantized networks. Compared with full-precision
parameters (i.e., 32-bit floating numbers), low-bit values are selected from a
much smaller set. For example, there are only 16 possibilities in 4-bit space.
Thus, we present to regard the discrete weights in an arbitrary quantized
neural network as searchable variables, and utilize a differential method to
search them accurately. In particular, each weight is represented as a
probability distribution over the discrete value set. The probabilities are
optimized during training and the values with the highest probability are
selected to establish the desired quantized network. Experimental results on
benchmarks demonstrate that the proposed method is able to produce quantized
neural networks with higher performance over the state-of-the-art methods on
both image classification and super-resolution tasks.

本文提出了通过利用微分方法来搜索离散权重的概率分布并在训练过程中进行优化，以建立所需的量化神经网络，该方法在图像分类和超分辨率任务上表现明显优于现有的最先进方法。