Benchmark Suite

Shootout(
    prob, setups; appxsol = nothing, names = nothing,
    error_estimate = :final, numruns = 20, seconds = 2, kwargs...
)

Benchmark multiple solver configurations on one problem. Each entry of setups is a dictionary containing an :alg and any solver-specific keyword arguments. The result stores the solutions, errors, timings, efficiencies, and the configuration with the highest efficiency, where efficiency is 1 / (error * time).

Use appxsol as a numerical reference when the problem has no analytic solution. Additional keyword arguments are forwarded to every solve.

ShootoutSet(
    probs, setups; probaux = nothing, names = nothing,
    print_names = false, kwargs...
)

Run a Shootout for every problem in probs. probaux may contain one dictionary of per-problem keyword arguments for each problem; other keyword arguments are shared by every shootout.

WorkPrecision(
    prob, alg, abstols, reltols, dts = nothing;
    name = nothing, appxsol = nothing, error_estimate = :final,
    numruns = 20, seconds = 2, tags = Symbol[], auto_tags = true,
    timeout = nothing, kwargs...
)

Measure error and execution time for alg at corresponding absolute and relative tolerances. When dts is provided, its entries select a fixed time step for each tolerance pair. The result stores the measured errors, timings, solver statistics, and inputs for plotting a work-precision diagram.

Use appxsol as a numerical reference when the problem has no analytic solution. tags attaches metadata symbols used by filter_by_tags and the plot recipe. They are merged with auto_tags(alg) unless auto_tags = false. timeout gives a per-tolerance wall-clock budget in seconds; see WorkPrecisionSet. Additional keyword arguments are forwarded to solve.

WorkPrecisionSet(prob, abstols, reltols, setups; error_estimates = nothing,
    timeout = nothing, kwargs...)

Build one WorkPrecision result for each solver configuration in setups so their work-precision curves can be compared. Each setup is a dictionary containing an :alg and may override the shared tolerances or fixed step sizes with :abstols, :reltols, or :dts, or its legend entry with :name. Preset tags from auto_tags are attached by default and merged with any :tags entry. Set :auto_tags => false in a setup to use only its explicit tags. The resulting metadata is used by filter_by_tags, best_of_families, autoplot, and the plot recipe to build family and cross-family comparisons.

error_estimates requests several error metrics from a single run (for example [:final, :l2, :L2]), so plots for each metric can be drawn without re-solving; the computed metrics are reported by available_errors. error_estimate remains the metric the plot recipe defaults to.

timeout is a per-tolerance wall-clock budget in seconds. A solve is never interrupted, but once one exceeds the budget its repeated timing runs are skipped and its error and time are recorded as NaN, which the plot recipe drops.

get_sample_errors(
    prob::AbstractRODEProblem, setup, test_dt = nothing;
    numruns, solution_runs, appxsol_setup = nothing,
    sample_error_runs = 10^7, parallel_type = :none, kwargs...
)

Estimate an approximate 95% confidence half-width for the sampling error of an RODE solver setup. numruns may be one sample count or a collection of counts; the return value is respectively a scalar or a collection of sampling-error estimates. solution_runs controls the number of independent estimates used to measure their variation.

When an analytic solution is available, sample_error_runs controls the Monte Carlo estimate of its expected endpoint. Otherwise, repeated numerical solution means provide the endpoint reference. Set parallel_type = :threads to parallelize the solution samples.

auto_tags(alg) -> Vector{Symbol}

Return preset metadata tags derived from the public traits and type hierarchy of alg. Differential-equation algorithms receive an order tag such as :order_5 (or :order_3_2 for order 3 // 2) when their order is defined, plus :adaptive or :fixed_step. When OrdinaryDiffEqCore is loaded, the result also describes known implicitness, families, and structural traits such as :rosenbrock, :firk, :sdirk, :split, and :multistep.

Algorithms without applicable traits return an empty vector. WorkPrecision and WorkPrecisionSet merge these presets with explicit tags by default.

get_tags(wp_set::WorkPrecisionSet) -> Vector{Vector{Symbol}}

Return the tags of each entry of wp_set, in order. Tags come from the :tags entry of the corresponding setup dictionary.

unique_tags(wp_set::WorkPrecisionSet) -> Vector{Symbol}

Return every tag used by any entry of wp_set, sorted and deduplicated.

filter_by_tags(wp_set::WorkPrecisionSet, tags::Symbol...) -> WorkPrecisionSet

Return the entries of wp_set tagged with every one of tags (AND logic), as a new WorkPrecisionSet. Passing no tags returns wp_set unchanged.

exclude_by_tags(wp_set::WorkPrecisionSet, tags::Symbol...) -> WorkPrecisionSet

Return the entries of wp_set carrying none of tags (OR logic on the exclusion), as a new WorkPrecisionSet. Passing no tags returns wp_set unchanged.

merge_wp_sets(sets::WorkPrecisionSet...) -> WorkPrecisionSet

Concatenate the entries of several WorkPrecisionSets into one. The tolerances, problem and error estimates are taken from the first set, so merging results computed on different problems or tolerance grids gives a set whose metadata describes only the first of them.

available_errors(wp_set::WorkPrecisionSet) -> Vector{Symbol}

Return the error estimates computed for wp_set, i.e. the error_estimates requested from WorkPrecisionSet or, when none were, the single error_estimate. Each one is a valid x for the plot recipe.

wp_area(wp::WorkPrecision) -> Float64

Trapezoidal area under the log₁₀(time)-vs-log₁₀(error) curve of wp, using the tolerances that produced a usable (positive, non-NaN) error and time. Lower is better. Returns Inf when fewer than two such points exist.

The area grows with the width of the error range covered, so it only compares methods that span a similar range; best_by_tag normalizes by that width instead.

best_by_tag(wp_set::WorkPrecisionSet, tag::Symbol; n = 1, metric = :area)
    -> WorkPrecisionSet

Return the n best-performing entries of wp_set tagged tag. With metric = :area (the only metric currently supported) entries are ranked first by how many tolerances produced a usable measurement — so a method that fails at tight tolerances does not win on the few points it survived — and then by their mean log₁₀ solve time over the log₁₀ error range they cover, i.e. wp_area normalized by that range.

best_of_families(wp_set::WorkPrecisionSet, family_tags; n = 1, metric = :area)
    -> WorkPrecisionSet

Combine the n best entries of each family in family_tags (see best_by_tag) into one WorkPrecisionSet for a cross-family comparison. An entry belonging to several families is included once. Throws an ArgumentError when no entry carries any of family_tags.

with_autodiff_variants(setups; ad_backends, tag_prefix = :autodiff)
    -> Vector{Dict{Symbol, Any}}

Expand setups with one copy per entry of ad_backends, each copy replacing :alg with the same algorithm rebuilt for that backend. Setups whose algorithm takes no autodiff argument (explicit Runge-Kutta methods, say) are passed through unexpanded.

The originals are kept and tagged $(tag_prefix)_default; each variant is tagged with the lowercased backend name, e.g. :autodiff_forwarddiff for AutoForwardDiff(), and named after its backend so the legends stay readable. Existing tags are preserved and the input setups are not mutated, so the result plots as an AD comparison via plot(wp_set, tags = [:autodiff_forwarddiff]).

autoplot(wp_set::WorkPrecisionSet; families = nothing, reference_tags = nothing,
    best_n = 2) -> Dict{String, WorkPrecisionSet}

Split one tagged WorkPrecisionSet into the subsets a benchmark page usually plots, without re-solving anything:

• "family_<tag>" for each family in families, holding that family's entries plus any entry matching reference_tags
• "best_of_families", the best_of_families selection of the best_n best entries per family, again alongside the reference entries
• "all", the full input set

Each value plots directly with plot(subset). When families is not given it is taken from the tags in use, dropping tags carried by more than 80% of the entries, since those describe the benchmark as a whole (:stiff, :nonstiff) rather than a family.