EnsembleGPUArray

EnsembleGPUArray(backend, cpu_offload = 0.2)

An EnsembleArrayAlgorithm that uses one kernel per trajectory while storing the trajectories in a batched array. This is the appropriate choice when the ODE solver or right-hand side cannot be compiled into one fused EnsembleGPUKernel solve.

Fields

• backend: the KernelAbstractions backend used for the batched computation.
• cpu_offload: the fraction of trajectories solved on the CPU. The default is 0.2.

Arguments

• backend: a KernelAbstractions backend, such as CUDA.CUDABackend() or KernelAbstractions.CPU().
• cpu_offload: the fraction of trajectories to offload to CPU execution. The two-argument constructor stores this value as a Float64; the one-argument constructor defaults to 0.2.

Returns

An EnsembleGPUArray algorithm selector.

Throws

The constructor itself does not validate backend support. Unsupported right-hand sides, callbacks, or solver combinations throw when the ensemble is solved.

Limitations

EnsembleGPUArray requires being able to generate a kernel for f using KernelAbstractions.jl and solving the resulting ODE defined over CuArray input types. This introduces the following limitations on its usage:

• Not all standard Julia f functions are allowed. Only Julia f functions which are capable of being compiled into a GPU kernel are allowed. This notably means that certain features of Julia can cause issues inside of kernel, like:

  • Allocating memory (building arrays)
  • Linear algebra (anything that calls BLAS)
  • Broadcast

• Not all ODE solvers are allowed, only those from OrdinaryDiffEq.jl. The tested feature set from OrdinaryDiffEq.jl includes:

  • Explicit Runge-Kutta methods
  • Implicit Runge-Kutta methods
  • Rosenbrock methods
  • DiscreteCallbacks and ContinuousCallbacks

• Stiff ODEs require the analytical solution of every derivative function it requires. For example, Rosenbrock methods require the Jacobian and the gradient with respect to time, and so these two functions are required to be given. Note that they can be generated by the ModelingToolkit.jl approach.

• To use multiple GPUs over clusters, one must manually set up one process per GPU. See the multi-GPU tutorial for more details.

Examples

using DiffEqGPU, CUDA, OrdinaryDiffEq
function lorenz(du, u, p, t)
    du[1] = p[1] * (u[2] - u[1])
    du[2] = u[1] * (p[2] - u[3]) - u[2]
    du[3] = u[1] * u[2] - p[3] * u[3]
    return
end

u0 = Float32[1.0; 0.0; 0.0]
tspan = (0.0f0, 100.0f0)
p = [10.0f0, 28.0f0, 8 / 3.0f0]
prob = ODEProblem(lorenz, u0, tspan, p)
prob_func = (prob, ctx) -> remake(prob, p = rand(Float32, 3) .* p)
monteprob = EnsembleProblem(prob; prob_func, safetycopy = false)
@time sol = solve(
    monteprob, Tsit5(), EnsembleGPUArray(CUDADevice()),
    trajectories = 10_000, saveat = 1.0f0
)

EnsembleCPUArray()

An EnsembleArrayAlgorithm which utilizes the CPU kernels to parallelize each ODE solve with their separate ODE integrator on each kernel. This method is meant to be a debugging counterpart to EnsembleGPUArray, having the same behavior and using the same KernelAbstractions.jl process to build the combined ODE, but without the restrictions of f being a GPU-compatible kernel function.

It is unlikely that this method is useful beyond library development and debugging, as almost any case should be faster with EnsembleThreads or EnsembleDistributed.

Returns

An EnsembleCPUArray algorithm selector for the CPU-backed array ensemble path.

Examples

ensemblealg = EnsembleCPUArray()
solve(ensemble_prob, Tsit5(), ensemblealg; trajectories = 100)