Dynamical System Variants

baideep2019 introduced Discrete Deep Equilibrium Models which drives a Discrete Dynamical System to its steady-state. pal2022mixing extends this framework to Continuous Dynamical Systems which converge to the steady-stable in a more stable fashion. For a detailed discussion refer to pal2022mixing.

Continuous DEQs

DeepEquilibriumNetworks.ContinuousDEQSolverType
ContinuousDEQSolver(alg=OrdinaryDiffEq.VCABM3(); mode::Symbol=:rel_deq_default,
                    abstol=1f-8, reltol=1f-8, abstol_termination=1f-8,
                    reltol_termination=1f-8, tspan=Inf32)

Solver for Continuous DEQ Problem (pal2022mixing). Similar to DynamicSS but provides more flexibility needed for solving DEQ problems.

Arguments

  • alg: Algorithm to solve the ODEProblem. (Default: VCABM3())
  • mode: Termination Mode of the solver. See below for a description of the various termination conditions (Default: :rel_deq_default)
  • abstol: Absolute tolerance for time stepping. (Default: 1f-8)
  • reltol: Relative tolerance for time stepping. (Default: 1f-8)
  • abstol_termination: Absolute tolerance for termination. (Default: 1f-8)
  • reltol_termination: Relative tolerance for termination. (Default: 1f-8)
  • tspan: Time span. Users should not change this value, instead control termination through maxiters in solve (Default: Inf32)

See also: DiscreteDEQSolver

Discrete DEQs

DeepEquilibriumNetworks.DiscreteDEQSolverType
DiscreteDEQSolver(alg=LimitedMemoryBroydenSolver(); mode::Symbol=:rel_deq_default,
                  abstol_termination::T=1.0f-8, reltol_termination::T=1.0f-8)

Solver for Discrete DEQ Problem (baideep2019). Similar to SSrootfind but provides more flexibility needed for solving DEQ problems.

Arguments

  • alg: Algorithm to solve the Nonlinear Problem. (Default: LimitedMemoryBroydenSolver)
  • mode: Termination Mode of the solver. See below for a description of the various termination conditions. (Default: :rel_deq_default)
  • abstol_termination: Absolute tolerance for termination. (Default: 1f-8)
  • reltol_termination: Relative tolerance for termination. (Default: 1f-8)

See also: ContinuousDEQSolver

Termination Conditions

Termination on Absolute Tolerance

  • :abs: Terminates if $all \left( | \frac{\partial u}{\partial t} | \leq abstol \right)$
  • :abs_norm: Terminates if $\| \frac{\partial u}{\partial t} \| \leq abstol$
  • :abs_deq_default: Essentially abs_norm + terminate if there has been no improvement for the last 30 steps + terminate if the solution blows up (diverges)
  • :abs_deq_best: Same as :abs_deq_default but uses the best solution found so far, i.e. deviates only if the solution has not converged

Termination on Relative Tolerance

  • :rel: Terminates if $all \left(| \frac{\partial u}{\partial t} | \leq reltol \times | u | \right)$
  • :rel_norm: Terminates if $\| \frac{\partial u}{\partial t} \| \leq reltol \times \| \frac{\partial u}{\partial t} + u \|$
  • :rel_deq_default: Essentially rel_norm + terminate if there has been no improvement for the last 30 steps + terminate if the solution blows up (diverges)
  • :rel_deq_best: Same as :rel_deq_default but uses the best solution found so far, i.e. deviates only if the solution has not converged

Termination using both Absolute and Relative Tolerances

  • :norm: Terminates if $\| \frac{\partial u}{\partial t} \| \leq reltol \times \| \frac{\partial u}{\partial t} + u \|$ & $\| \frac{\partial u}{\partial t} \| \leq abstol$
  • fallback: Check if all values of the derivative is close to zero wrt both relative and absolute tolerance. This is usable for small problems but doesn't scale well for neural networks, and should be avoided unless absolutely necessary