Optimisers.jl

Installation: OptimizationOptimisers.jl

To use this package, install the OptimizationOptimisers package:

import Pkg;
Pkg.add("OptimizationOptimisers");

In addition to the optimisation algorithms provided by the Optimisers.jl package this subpackage also provides the Sophia optimisation algorithm.

Reexported Optimisers.jl API

using OptimizationOptimisers brings Optimisers.jl's rules into scope, so that solve(prob, Adam(0.05)) works without a separate using Optimisers. These names are owned and documented by Optimisers.jl; this package only re-exports them.

• Gradient descent rules: Descent, Momentum, Nesterov, Rprop
• Adaptive rules: RMSProp, Adam, RAdam, AdaMax, OAdam, AdaGrad, AdaDelta, AMSGrad, NAdam, AdamW, AdaBelief, Lion
• Legacy all-caps spellings kept for compatibility: ADAM, ADAMW, RADAM, OADAM, NADAM, ADAGrad, ADADelta
• Gradient modifiers and combinators: ClipGrad, ClipNorm, WeightDecay, SignDecay, AccumGrad, OptimiserChain
• The Optimisers module itself; the rule supertype is Optimisers.AbstractRule

Optimisers' own optimiser-driving interface — Optimisers.setup, Optimisers.update, Optimisers.update!, Optimisers.apply!, Optimisers.destructure, Optimisers.trainables — is deliberately not re-exported: solve drives the rule for you, and Optimisers.init would collide with the SciML init.

Anything else from Optimisers.jl must be imported from Optimisers directly.

List of optimizers

• Optimisers.Descent: Classic gradient descent optimizer with learning rate

  • solve(problem, Descent(η))

  • η is the learning rate

  • Defaults:

    • η = 0.1

• Optimisers.Momentum: Classic gradient descent optimizer with learning rate and momentum

  • solve(problem, Momentum(η, ρ))

  • η is the learning rate

  • ρ is the momentum

  • Defaults:

    • η = 0.01
    • ρ = 0.9

• Optimisers.Nesterov: Gradient descent optimizer with learning rate and Nesterov momentum

  • solve(problem, Nesterov(η, ρ))

  • η is the learning rate

  • ρ is the Nesterov momentum

  • Defaults:

    • η = 0.01
    • ρ = 0.9

• Optimisers.RMSProp: RMSProp optimizer

  • solve(problem, RMSProp(η, ρ))

  • η is the learning rate

  • ρ is the momentum

  • Defaults:

    • η = 0.001
    • ρ = 0.9

• Optimisers.Adam: Adam optimizer

  • solve(problem, Adam(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)

• Optimisers.RAdam: Rectified Adam optimizer

  • solve(problem, RAdam(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)

• Optimisers.OAdam: Optimistic Adam optimizer

  • solve(problem, OAdam(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.5, 0.999)

• Optimisers.AdaMax: AdaMax optimizer

  • solve(problem, AdaMax(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)

• Optimisers.ADAGrad: ADAGrad optimizer

  • solve(problem, ADAGrad(η))

  • η is the learning rate

  • Defaults:

    • η = 0.1

• Optimisers.ADADelta: ADADelta optimizer

  • solve(problem, ADADelta(ρ))

  • ρ is the gradient decay factor

  • Defaults:

    • ρ = 0.9

• Optimisers.AMSGrad: AMSGrad optimizer

  • solve(problem, AMSGrad(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)

• Optimisers.NAdam: Nesterov variant of the Adam optimizer

  • solve(problem, NAdam(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)

• Optimisers.AdamW: AdamW optimizer

  • solve(problem, AdamW(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • decay is the decay to weights

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)
    • decay = 0

• Optimisers.ADABelief: ADABelief variant of Adam

  • solve(problem, ADABelief(η, β::Tuple))

  • η is the learning rate

  • β::Tuple is the decay of momentums

  • Defaults:

    • η = 0.001
    • β::Tuple = (0.9, 0.999)