Multistart optimization with EnsembleProblem

The EnsembleProblem in SciML serves as a common interface for running a problem on multiple sets of initializations. In the context of optimization, this is useful for performing multistart optimization.

This can be useful for complex, low dimensional problems. We demonstrate this, again, on the rosenbrock function.

We first execute a single local optimization with BFGS and maxiters=5:

using OptimizationBase, OptimizationOptimJL, Random
using SciMLBase, ADTypes, ForwardDiff

Random.seed!(100)

rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)

optf = OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff())
prob = OptimizationProblem(optf, x0, [1.0, 100.0])
@time sol1 = solve(prob, BFGS(), maxiters = 5)

@show sol1.objective

0.2557901706445054

This results is compared to a multistart approach with 4 random initial points:

x0s = [x0, x0 .+ rand(2), x0 .+ rand(2), x0 .+ rand(2)]
function prob_func(prob, ctx)
    remake(prob, u0 = x0s[ctx.sim_id])
end

ensembleprob = EnsembleProblem(prob; prob_func)
@time sol = solve(ensembleprob, BFGS(),
    EnsembleThreads(), trajectories = 4, maxiters = 5)
@show minimum(s.objective for s in sol.u)

0.017896203029448516

With the same number of iterations (5) we get a much lower (1/100th) objective value by using multiple initial points. The initialization strategy used here was a pretty trivial one but approaches based on Quasi-Monte Carlo sampling should be typically more effective.