Adaptive Loss Functions

AbstractAdaptiveLoss

Abstract supertype for adaptive loss-weighting strategies used by PhysicsInformedNN.

NonAdaptiveLoss(; pde_loss_weights = 1.0,
                  bc_loss_weights = 1.0,
                  additional_loss_weights = 1.0)

A way of loss weighting the components of the loss function in the total sum that does not change during optimization

GradientScaleAdaptiveLoss(reweight_every;
                          weight_change_inertia = 0.9,
                          pde_loss_weights = 1.0,
                          bc_loss_weights = 1.0,
                          additional_loss_weights = 1.0)

A way of adaptively reweighting the components of the loss function in the total sum such that BCi loss weights are scaled by the exponential moving average of max(|∇pdeloss|) / mean(|∇bciloss|)).

Positional Arguments

• reweight_every: how often to reweight the BC loss functions, measured in iterations. Reweighting is somewhat expensive since it involves evaluating the gradient of each component loss function,

Keyword Arguments

• weight_change_inertia: a real number that represents the inertia of the exponential moving average of the BC weight changes,

References

Understanding and mitigating gradient pathologies in physics-informed neural networks Sifan Wang, Yujun Teng, Paris Perdikaris https://arxiv.org/abs/2001.04536v1

With code reference: https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs

MiniMaxAdaptiveLoss(reweight_every;
                    pde_max_optimiser = OptimizationOptimisers.Adam(1e-4),
                    bc_max_optimiser = OptimizationOptimisers.Adam(0.5),
                    pde_loss_weights = 1, bc_loss_weights = 1,
                    additional_loss_weights = 1)

A way of adaptively reweighting the components of the loss function in the total sum such that the loss weights are maximized by an internal optimizer, which leads to a behavior where loss functions that have not been satisfied get a greater weight.

Positional Arguments

• reweight_every: how often to reweight the PDE and BC loss functions, measured in iterations. Reweighting is cheap since it re-uses the value of loss functions generated during the main optimization loop.

Keyword Arguments

• pde_max_optimiser: a OptimizationOptimisers optimiser that is used internally to maximize the weights of the PDE loss functions.
• bc_max_optimiser: a OptimizationOptimisers optimiser that is used internally to maximize the weights of the BC loss functions.

References

Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism Levi McClenny, Ulisses Braga-Neto https://arxiv.org/abs/2009.04544

SoftAdaptAdaptiveLoss(reweight_every;
                      α = 0.1,
                      pde_loss_weights = 1.0,
                      bc_loss_weights = 1.0,
                      additional_loss_weights = 1.0)

An adaptive loss weighting strategy based on the relative rate of change of each loss component. Weights are assigned proportionally via a softmax over the normalised loss rates, so components that are growing faster receive a larger weight.

No gradient computations are required; reweighting cost is O(N) in the number of loss terms, making it cheaper than GradientScaleAdaptiveLoss.

Positional Arguments

• reweight_every: how often (in iterations) to update the loss weights.

Keyword Arguments

• α: temperature parameter controlling the sharpness of the softmax (default 0.1). Higher values make the weighting more aggressive.

Algorithm

rate_i(t) = (L_i(t) - L_i(t-1)) / (L_i(t-1) + ε)
λ_i(t)    = softmax(α · rate(t)) × N

References

Heydari, A. A., Thompson, C. A., & Mehmood, A. (2019). SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions. arXiv:1912.12355. https://arxiv.org/abs/1912.12355

ReLoBRaLoAdaptiveLoss(reweight_every;
                      α = 1.0,
                      β = 0.9,
                      pde_loss_weights = 1.0,
                      bc_loss_weights = 1.0,
                      additional_loss_weights = 1.0)

Relative Loss Balancing with Random Lookback (ReLoBRaLo). Adaptively reweights loss components by comparing their current value to a randomly chosen past reference — either the initial loss or the most recent checkpoint — controlled by the lookback probability β.

This makes the method more robust to short-term loss oscillations than purely incremental strategies like SoftAdaptAdaptiveLoss. No gradient computations are required.

Positional Arguments

• reweight_every: how often (in iterations) to update the loss weights.

Keyword Arguments

• α: temperature parameter for the softmax (default 1.0).
• β: probability of using the previous checkpoint as reference instead of the initial losses (default 0.9). Setting β = 0 always uses the initial losses; β = 1 always uses the most recent checkpoint.

Algorithm

ρ    ~ Bernoulli(β)
t₀   = ρ · t_prev + (1 - ρ) · t_init
λ_i  = softmax(α · L_i(t) / (L_i(t₀) + ε)) × N

References

Bischof, R., & Kraus, M. (2021). Multi-Objective Loss Balancing for Physics-Informed Deep Learning. arXiv:2110.09813. https://arxiv.org/abs/2110.09813