SimpleOptimization.jl

SimpleBFGS()

A lightweight, loop-unrolled BFGS optimization algorithm. This algorithm is designed for small-scale unconstrained optimization problems where low overhead is critical.

Description

SimpleBFGS implements the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method. It builds an approximation to the inverse Hessian matrix using gradient information, achieving superlinear convergence for smooth objective functions.

Internally, it wraps SimpleBroyden from SimpleNonlinearSolve.jl to find the root of the gradient (i.e., the stationary point of the objective).

Example

using SimpleOptimization, Optimization, ForwardDiff

rosenbrock(x, p) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
x0 = zeros(2)
optf = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff())
prob = OptimizationProblem(optf, x0)
sol = solve(prob, SimpleBFGS())

SimpleLBFGS(; threshold::Union{Val, Int} = Val(10),
            linesearch = StrongWolfeLineSearch(; maxiters = 50, zoom_maxiters = 50))

A lightweight, loop-unrolled Limited-memory BFGS (L-BFGS) optimization algorithm. This algorithm is designed for small-scale optimization problems where low overhead is critical.

Arguments

• threshold: The number of past iterations to store for approximating the inverse Hessian. Default is Val(10). Can be specified as either a Val type for compile-time optimization or an Int.
• linesearch: A LineSearch.StrongWolfeLineSearch used for step-size selection.

Description

SimpleLBFGS uses a limited-memory approximation to the BFGS update, storing only the last threshold iterations of gradient information. This makes it memory-efficient for problems with many variables while still achieving superlinear convergence.

Minimizes the objective with a Strong Wolfe line search. Supports box constraints (lb/ub) via projection. Works with any u0; a statically sized u0 (e.g. SVector) keeps the solve allocation-free.

ReturnCode.Success when the projected gradient meets the tolerance, ReturnCode.MaxIters when the iteration limit is hit, and ReturnCode.Failure on a non-finite iterate or a failed line search.

Example

using SimpleOptimization, Optimization, ForwardDiff

rosenbrock(x, p) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
x0 = zeros(2)
optf = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff())
prob = OptimizationProblem(optf, x0)
sol = solve(prob, SimpleLBFGS())

SimpleGradientDescent(; eta = 0.01)

A lightweight gradient descent optimization algorithm. This algorithm is designed for small-scale unconstrained optimization problems where low overhead is critical.

Arguments

• eta: The learning rate (step size). Default is 0.01.

Description

SimpleGradientDescent implements the steepest descent method, updating the iterate via x_{k+1} = x_k - eta * gradient(f, x_k) at each step. While it has only linear convergence, it is the simplest first-order method and is useful as a baseline or for problems where quasi-Newton overhead is undesirable.

Example

using SimpleOptimization, Optimization, ForwardDiff

rosenbrock(x, p) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
x0 = zeros(2)
optf = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff())
prob = OptimizationProblem(optf, x0)
sol = solve(prob, SimpleGradientDescent(; eta = 0.001), maxiters = 10000)

SimpleNewton()

A lightweight Newton optimization algorithm. This algorithm is designed for small-scale unconstrained optimization problems where quadratic convergence is desired.

Description

SimpleNewton implements Newton's method for optimization, which finds a stationary point by solving the system gradient(f, x) = 0 using Newton-Raphson iteration. This requires computing the Hessian (via automatic differentiation of the gradient) and gives quadratic convergence near the solution for smooth objective functions.

Internally, it wraps SimpleNewtonRaphson from SimpleNonlinearSolve.jl to find the root of the gradient. The Hessian is computed automatically by SimpleNewtonRaphson's internal AD applied to the gradient function.

Example

using SimpleOptimization, Optimization, ForwardDiff

rosenbrock(x, p) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
x0 = [0.5, 0.5]
optf = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff())
prob = OptimizationProblem(optf, x0)
sol = solve(prob, SimpleNewton())

SimpleSOAP(; eta=3e-3, beta=(0.95, 0.95), shampoo_beta=-1.0, epsilon=1e-8,
             freq=10, max_dim=10000, weight_decay=0.01)

SOAP optimizer (ShampoO with Adam in the Preconditioner's eigenbasis). For matrix-valued parameters, runs AdamW in the eigenbasis of Shampoo's preconditioner. For vector-valued parameters, falls back to standard AdamW.

Based on "SOAP: Improving and Stabilizing Shampoo using Adam" (https://arxiv.org/abs/2409.11321).

Arguments

• eta: learning rate (default: 3e-3)
• beta: (β₁, β₂) for Adam momentum and second moment (default: (0.95, 0.95))
• shampoo_beta: separate β for preconditioner EMA; if < 0, uses β₂ (default: -1)
• epsilon: numerical stability constant (default: 1e-8)
• freq: how often to recompute eigenbasis (default: 10)
• max_dim: dimensions larger than this use identity rotation (default: 10000)
• weight_decay: decoupled weight decay, applied as lr * wd (default: 0.01)

Example

using SimpleOptimization, ForwardDiff

f(x, p) = sum(abs2, x .- p)
W0 = randn(8, 8)
optf = OptimizationFunction(f, AutoForwardDiff())
prob = OptimizationProblem(optf, W0, ones(8, 8))
sol = solve(prob, SimpleSOAP(), maxiters = 500)