Models

AdditiveEIESN(in_dims, res_dims, out_dims, activation=tanh_fast;
        input_activation=identity,
        use_bias=true,
        exc_recurrence_scale=0.9, inh_recurrence_scale=0.5, exc_output_scale=1.0,
        inh_output_scale=1.0,init_reservoir=rand_sparse,
        init_input=scaled_rand, init_bias=zeros32,
        init_state=randn32,
        readout_activation=identity,
        state_modifiers=(),
        readout_in_dims=nothing,
        kwargs...)

Excitatory-Inhibitory Echo State Network (EIESN) with additive input (Panahi et al., 2025).

This model wraps AdditiveEIESNCell, where the input is added linearly outside the non-linearity with optional bias terms.

Equations

\[\begin{aligned} \mathbf{x}(t) &= b_{\mathrm{ex}} \, \phi_{\mathrm{ex}}\!\left( a_{\mathrm{ex}} \mathbf{A} \mathbf{x}(t-1) + \mathbf{\beta}_{\mathrm{ex}}\right) - b_{\mathrm{inh}} \, \phi_{\mathrm{inh}}\!\left( a_{\mathrm{inh}} \mathbf{A} \mathbf{x}(t-1) + \mathbf{\beta}_{\mathrm{inh}}\right) + g\!\left( \mathbf{W}_{\mathrm{in}} \mathbf{u}(t) + \mathbf{\beta}_{\mathrm{in}}\right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for AdditiveEIESNCell). Can be a single function or a Tuple(excitatory, inhibitory). Default: tanh_fast.

Keyword arguments

• input_activation: The non-linear function $g$ applied to the input. Default: identity.
• use_bias: Enable/disable bias vectors. Default: true.
• exc_recurrence_scale: Excitatory recurrence scaling factor. Default: 0.9.
• inh_recurrence_scale: Inhibitory recurrence scaling factor. Default: 0.5.
• exc_output_scale: Excitatory output scaling factor. Default: 1.0.
• inh_output_scale: Inhibitory output scaling factor. Default: 1.0.
• init_reservoir: Initializer for the reservoir matrix. Default: rand_sparse.
• init_input: Initializer for the input matrix. Default: scaled_rand.
• init_bias: Initializer for the bias vectors. Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• readout_activation: Activation for the linear readout. Default: identity.
• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal AdditiveEIESNCell.
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout.

States

• reservoir — states for the internal AdditiveEIESNCell (e.g. rng).
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

DeepESN(in_dims, res_dims, out_dims,
        activation=tanh; depth=2, leak_coefficient=1.0, init_reservoir=rand_sparse,
        init_input=scaled_rand, init_bias=zeros32, init_state=randn32,
        use_bias=false, state_modifiers=(), readout_activation=identity,
        readout_in_dims=nothing)

Deep Echo State Network (Gallicchio and Micheli, 2017).

DeepESN composes, for L = length(res_dims) layers:

1. a sequence of stateful ESNCell with widths res_dims[ℓ],
2. zero or more per-layer state_modifiers[ℓ] applied to the layer's state, and
3. a final LinearReadout from the last layer's features to the output.

Equations

\[\begin{aligned} \mathbf{x}^{(1)}(t) &= (1-\alpha_1)\, \mathbf{x}^{(1)}(t-1) + \alpha_1\, \phi_1\!\left(\mathbf{W}^{(1)}_{\text{in}}\, \mathbf{u}(t) + \mathbf{W}^{(1)}_r\, \mathbf{x}^{(1)}(t-1) + \mathbf{b}^{(1)} \right), \\ \mathbf{u}^{(1)}(t) &= \mathrm{Mods}_1\!\left(\mathbf{x}^{(1)}(t)\right), \\ \mathbf{x}^{(\ell)}(t) &= (1-\alpha_\ell)\, \mathbf{x}^{(\ell)}(t-1) + \alpha_\ell\, \phi_\ell\!\left(\mathbf{W}^{(\ell)}_{\text{in}}\, \mathbf{u}^{(\ell-1)}(t) + \mathbf{W}^{(\ell)}_r\, \mathbf{x}^{(\ell)}(t-1) + \mathbf{b}^{(\ell)} \right), \quad \ell = 2,\dots,L, \\ \mathbf{u}^{(\ell)}(t) &= \mathrm{Mods}_\ell\!\left(\mathbf{x}^{(\ell)}(t)\right), \quad \ell = 2,\dots,L, \\ \mathbf{y}(t) &= \rho\!\left(\mathbf{W}_{\text{out}}\, \mathbf{u}^{(L)}(t) + \mathbf{b}_{\text{out}} \right). \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Vector of reservoir (hidden) dimensions per layer; its length sets the depth L.
• out_dims: Output dimension.
• activation: Reservoir activation(s). Either a single function (broadcast to all layers) or a vector/tuple of length L. Default: tanh.

Keyword arguments

Per-layer reservoir options (passed to each ESNCell):

• leak_coefficient: Leak rate(s) α_ℓ ∈ (0,1]. Scalar or length-L collection. Default: 1.0.
• init_reservoir: Initializer(s) for W_res^{(ℓ)}. Scalar or length-L. Default: rand_sparse.
• init_input: Initializer(s) for W_in^{(ℓ)}. Scalar or length-L. Default: scaled_rand.
• init_bias: Initializer(s) for reservoir bias (used iff use_bias[ℓ]=true). Scalar or length-L. Default: zeros32.
• init_state: Initializer(s) used when an external state is not provided. Scalar or length-L. Default: randn32.
• use_bias: Whether each reservoir uses a bias term. Boolean scalar or length-L. Default: false.

Depth:

• depth: Number of reservoir layers. Only used when res_dims is given as a single integer (the depth is then depth layers of that width); it is ignored when res_dims is a vector, whose length already sets the depth L. Default: 2.

Composition:

• state_modifiers: Per-layer modifier(s) applied to each layer’s state before it feeds into the next layer (and the readout for the last layer). Accepts nothing, a single layer, a vector/tuple of length L, or per-layer collections. Defaults to no modifiers.
• readout_in_dims: Final readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the final linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple) containing states for all cells, modifiers, and readout.

Parameters

• cells :: NTuple{L,NamedTuple} — parameters for each ESNCell, including:
  • input_matrix :: (res_dims[ℓ] × in_size[ℓ]) — W_in^{(ℓ)}
  • reservoir_matrix :: (res_dims[ℓ] × res_dims[ℓ]) — W_res^{(ℓ)}
  • bias :: (res_dims[ℓ],) — present only if use_bias[ℓ]=true
• state_modifiers :: NTuple{L,Tuple} — per-layer tuples of modifier parameters (empty tuples if none).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims[L]) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

• cells :: NTuple{L,NamedTuple} — states for each ESNCell.
• state_modifiers :: NTuple{L,Tuple} — per-layer tuples of modifier states.
• readout — states for LinearReadout.

DelayESN(in_dims, res_dims, out_dims, activation=tanh;
             num_input_delays=1, input_stride=1,
             num_state_delays=1, state_stride=1,
             leak_coefficient=1.0, init_reservoir=rand_sparse,
             init_input=scaled_rand, init_bias=zeros32,
             init_state=randn32, use_bias=false,
             state_modifiers=(), readout_activation=identity,
             readout_in_dims=nothing)

Echo State Network with both input and state delays (Fleddermann et al., 2025).

DelayESN composes:

1. an internal DelayLayer applied to the input signal,
2. a stateful ESNCell (reservoir) receiving the augmented input,
3. a second internal DelayLayer applied to the reservoir state,
4. zero or more additional state_modifiers applied after the state delay, and
5. a LinearReadout mapping the final feature vector to outputs.

At each time step, the input u(t) is expanded into a delay-coordinate vector u_d(t). This drives the reservoir to produce state x(t). Finally, x(t) is expanded into a state delay-coordinate vector x_d(t) before passing to modifiers and readout.

Equations

\[\begin{aligned} \mathbf{u}_{\mathrm{d}}(t) &= \begin{bmatrix} \mathbf{u}(t) \\ \mathbf{u}(t-s_{in}) \\ \vdots \\ \mathbf{u}\!\bigl(t-D_{in}s_{in}\bigr) \end{bmatrix}, \qquad D_{in}=\text{num\_input\_delays},\ \ s_{in}=\text{input\_stride}, \\ \mathbf{x}(t) &= (1-\alpha)\, \mathbf{x}(t-1) + \alpha\, \phi\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}_{\mathrm{d}}(t) + \mathbf{W}_r\, \mathbf{x}(t-1) + \mathbf{b} \right), \\ \mathbf{x}_{\mathrm{d}}(t) &= \begin{bmatrix} \mathbf{x}(t) \\ \mathbf{x}(t-s_{st}) \\ \vdots \\ \mathbf{x}\!\bigl(t-D_{st}s_{st}\bigr) \end{bmatrix}, \qquad D_{st}=\text{num\_state\_delays},\ \ s_{st}=\text{state\_stride}, \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left( \mathbf{x}_{\mathrm{d}}(t)\right), \\ \mathbf{y}(t) &= \rho\!\left(\mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

Reservoir (passed to ESNCell):

• leak_coefficient: Leak rate in (0, 1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used iff use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.

Input delay expansion:

• num_input_delays: Number of past input steps to include. The effective input to the reservoir has size (num_input_delays + 1) * in_dims. Default: 1.
• input_stride: Stride for the input delay buffer. Default: 1.

State delay expansion:

• num_state_delays: Number of past reservoir states to include. The readout receives a vector of size (num_state_delays + 1) * res_dims. Default: 1.
• state_stride: Stride for the state delay buffer. Default: 1.

Composition:

• state_modifiers: A layer or collection of layers applied to the delayed reservoir features before the readout. These run after the internal state DelayLayer. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• input_delay — parameters for the input DelayLayer.
• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × ((num_input_delays + 1) * in_dims)) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for:
  1. the internal state DelayLayer, and
  2. any user-provided modifier layers (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × ((num_state_delays + 1) * res_dims)) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

States

• input_delay — state for the input DelayLayer (buffer and clock).
• reservoir — states for the internal ESNCell (e.g. rng used to sample initial hidden states).
• state_modifiers — a Tuple with states for the internal state DelayLayer (its delay buffer and clock) and each additional modifier layer.
• readout — states for LinearReadout (typically empty).

EIESN(in_dims, res_dims, out_dims, activation=tanh_fast;
    use_bias=true,
    exc_recurrence_scale=0.9, inh_recurrence_scale=0.5, exc_output_scale=1.0,
    inh_output_scale=1.0, init_reservoir=rand_sparse,
    init_input=scaled_rand, init_bias=zeros32,
    init_state=randn32,
    readout_activation=identity,
    state_modifiers=(),
    readout_in_dims=nothing,
    kwargs...)

Excitatory-Inhibitory Echo State Network (EIESN) (Panahi et al., 2025).

This model wraps EIESNCell.

Equations

\[\begin{aligned} \mathbf{x}(t) &= b_{\mathrm{ex}} \, \phi_{\mathrm{ex}}\!\left( \mathbf{W}_{\mathrm{in}} \mathbf{u}(t) + a_{\mathrm{ex}} \mathbf{A} \mathbf{x}(t-1) + \mathbf{\beta}_{\mathrm{ex}}\right) - b_{\mathrm{inh}} \, \phi_{\mathrm{inh}}\!\left( \mathbf{W}_{\mathrm{in}} \mathbf{u}(t) + a_{\mathrm{inh}} \mathbf{A} \mathbf{x}(t-1) + \mathbf{\beta}_{\mathrm{inh}}\right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for EIESNCell). Can be a single function or a Tuple(excitatory, inhibitory). Default: tanh_fast.

Keyword arguments

• use_bias: Enable/disable bias vectors inside the reservoir. Default: true.
• exc_recurrence_scale: Excitatory recurrence scaling factor. Default: 0.9.
• inh_recurrence_scale: Inhibitory recurrence scaling factor. Default: 0.5.
• exc_output_scale: Excitatory output scaling factor. Default: 1.0.
• inh_output_scale: Inhibitory output scaling factor. Default: 1.0.
• init_reservoir: Initializer for the reservoir matrix. Default: rand_sparse.
• init_input: Initializer for the input matrix. Default: scaled_rand.
• init_bias: Initializer for the bias vectors. Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• readout_activation: Activation for the linear readout. Default: identity.
• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal EIESNCell.
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout.

States

• reservoir — states for the internal EIESNCell (e.g. rng).
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

ES2N(in_dims, res_dims, out_dims, activation=tanh;
    proximity=1.0, init_reservoir=rand_sparse, init_input=scaled_rand,
    init_bias=zeros32, init_state=randn32, use_bias=False(),
    state_modifiers=(), readout_activation=identity,
    init_orthogonal=orthogonal, readout_in_dims=nothing)

Edge of Stability Echo State Network (ES2N) (Ceni and Gallicchio, 2025).

Equations

\[\begin{aligned} \mathbf{x}(t) &= (1-\beta)\, \mathbf{O}\, \mathbf{x}(t-1) + \beta\, \phi\!\left(\mathbf{W}_{\text{in}} \mathbf{u}(t) + \mathbf{W}_r \mathbf{x}(t-1) + \mathbf{b} \right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

• proximity: proximity α ∈ (0,1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_orthogonal: Initializer for O. Default: [orthogonal].
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.
• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × in_dims) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • orthogonal_matrix :: (res_dims × res_dims) — O
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

• reservoir — states for the internal ES2NCell (e.g. rng used to sample initial hidden states).
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

ESN(in_dims, res_dims, out_dims, activation=tanh;
    leak_coefficient=1.0, init_reservoir=rand_sparse, init_input=scaled_rand,
    init_bias=zeros32, init_state=randn32, use_bias=false,
    state_modifiers=(), readout_activation=identity, readout_in_dims=nothing)

Echo State Network (Jaeger and Haas, 2004).

ESN composes:

1. a stateful ESNCell (reservoir),
2. zero or more state_modifiers applied to the reservoir state, and
3. a LinearReadout mapping reservoir features to outputs.

Equations

\[\begin{aligned} \mathbf{x}(t) &= (1-\alpha)\, \mathbf{x}(t-1) + \alpha\, \phi\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t) + \mathbf{W}_r\, \mathbf{x}(t-1) + \mathbf{b} \right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

Reservoir (passed to ESNCell):

• leakcoefficient: Leak rate α ∈ (0,1]. Can be a scalar (uniform leak) or a vector of size `outdims(heterogeneous leak rates). Default:1.0`.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.

Composition:

• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × in_dims) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

• reservoir — states for the internal ESNCell (e.g. rng used to sample initial hidden states).
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

EuSN(in_dims, res_dims, out_dims, activation=tanh;
    leak_coefficient=1.0, diffusion = 1.0, init_reservoir=rand_sparse, init_input=scaled_rand,
    init_bias=zeros32, init_state=randn32, use_bias=false,
    state_modifiers=(), readout_activation=identity, readout_in_dims=nothing)

Euler State Network (ESN) (Gallicchio, 2024).

Equations

\[\begin{aligned} \mathbf{x}(t) &= \mathbf{x}(t-1) + \varepsilon\, \phi\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t) + (\mathbf{W}_r - \gamma\, \mathbf{I})\, \mathbf{x}(t-1) + \mathbf{b} \right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

• leak_coefficient: Leak rate α ∈ (0,1]. Default: 1.0.
• diffusion: diffusion coefficient ∈ (0,1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.
• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × in_dims) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

• reservoir — states for the internal ESNCell (e.g. rng used to sample initial hidden states).
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

HybridESN(km, km_dims, in_dims, res_dims, out_dims, [activation];
    state_modifiers=(), readout_activation=identity,
    include_collect=true, readout_in_dims=nothing, kwargs...)

Hybrid Echo State Network (Pathak et al., 2018).

HybridESN composes:

1. a knowledge model km producing auxiliary features from the input,
2. a stateful ESNCell that receives the concatenated input [km(x(t)); x(t)],
3. zero or more state_modifiers applied to the reservoir state, and
4. a LinearReadout mapping the combined features [km(x(t)); h*(t)] to the output.

Arguments

• km: Knowledge model applied to the input (e.g. a physical model, neural submodule, or differentiable function). May be a WrappedFunction or any callable layer.
• km_dims: Output dimension of the knowledge model km.
• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

• leak_coefficient: Leak rate α ∈ (0,1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.
• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Total readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.
• include_collect: Whether the readout should include collection mode. Default: true.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• knowledge_model — parameters of the knowledge model km.
• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × (in_dims + km_dims)) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × (res_dims + km_dims)) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

Created by initialstates(rng, hesn):

• knowledge_model — states for the internal knowledge model.
• reservoir — states for the internal ESNCell.
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

RMNESN(in_dims, mem_dims, res_dims, out_dims, activation=tanh;
    init_memory_reservoir=simple_cycle(; cycle_weight=1),
    init_memory_input=scaled_rand,
    init_memory_bias=zeros32,
    init_memory_state=randn32,
    use_memory_bias=False(),
    init_reservoir=rand_sparse,
    init_input=scaled_rand,
    init_memory=scaled_rand,
    init_bias=zeros32,
    init_state=randn32,
    use_bias=False(),
    state_modifiers=(),
    readout_activation=identity,
    readout_in_dims=nothing)

Construct a Reservoir Memory Network Echo State Network (Gallicchio and Ceni, 2024).

Arguments

• in_dims: Input dimension.
• mem_dims: Linear memory reservoir dimension.
• res_dims: Nonlinear reservoir hidden state dimension.
• out_dims: Output dimension.
• activation: Activation function for the nonlinear reservoir. Default: tanh.

Keyword arguments

• init_memory_reservoir: Initializer for the memory reservoir recurrent matrix. Default: simple_cycle(; cycle_weight=1).
• init_memory_input: Initializer for the memory reservoir input matrix. Default: scaled_rand.
• init_memory_bias: Initializer for the memory reservoir bias. Default: zeros32.
• init_memory_state: Initializer used when an external memory state is not provided. Default: randn32.
• use_memory_bias: Whether the memory reservoir uses a bias term. Default: False().
• init_reservoir: Initializer for the nonlinear reservoir recurrent matrix. Default: rand_sparse.
• init_input: Initializer for the nonlinear reservoir input matrix. Default: scaled_rand.
• init_memory: Initializer for the matrix coupling the memory state into the nonlinear reservoir. Default: scaled_rand.
• init_bias: Initializer for the nonlinear reservoir bias, used iff use_bias=true. Default: zeros32.
• init_state: Initializer used when an external nonlinear reservoir state is not provided. Default: randn32.
• use_bias: Whether the nonlinear reservoir uses a bias term. Default: False().
• state_modifiers: A layer or collection of layers applied to the nonlinear reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state NamedTuple.

Parameters

• reservoir — parameters of the internal RMNCell, containing:
  • linear_reservoir — parameters of the memory ESNCell, including:
    • input_matrix :: (mem_dims × in_dims).
    • reservoir_matrix :: (mem_dims × mem_dims).
    • bias :: (mem_dims,) — present only if use_memory_bias=true.
  • nonlinear_reservoir — parameters of MemoryESNCell, including:
    • input_matrix :: (res_dims × in_dims).
    • reservoir_matrix :: (res_dims × res_dims).
    • memory_matrix :: (res_dims × mem_dims).
    • bias :: (res_dims,) — present only if use_bias=true.
• state_modifiers — a Tuple with parameters for each modifier layer; may be empty.
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims).
  • bias :: (out_dims,).

States

• reservoir — states for the internal RMNCell, containing:
  • linear_reservoir — states for the memory ESNCell.
  • nonlinear_reservoir — states for MemoryESNCell.
  • rng — random number generator state used to sample initial recurrent states.
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

See also

RMNCell, MemoryESNCell, ESNCell, LinearReadout

RMNResESN(in_dims, mem_dims, res_dims, out_dims, activation=tanh;
    init_memory_reservoir=simple_cycle(; cycle_weight=1),
    init_memory_input=scaled_rand,
    init_memory_bias=zeros32,
    init_memory_state=randn32,
    use_memory_bias=False(),
    init_reservoir=rand_sparse,
    init_input=scaled_rand,
    init_memory=scaled_rand,
    init_orthogonal=orthogonal,
    init_bias=zeros32,
    init_state=randn32,
    use_bias=False(),
    alpha=1.0, beta=1.0,
    state_modifiers=(),
    readout_activation=identity,
    readout_in_dims=nothing)

Residual Reservoir Memory Network (Ceni et al., 2025). Combines a linear memory reservoir with a residual nonlinear reservoir that additionally consumes the memory state, following the same composition pattern as RMNESN but using a MemoryResESNCell as the nonlinear reservoir.

Equations

\[\begin{aligned} \mathbf{m}(t) &= \mathbf{W}_{\text{in}}^{m}\, \mathbf{u}(t) + \mathbf{C}\, \mathbf{m}(t-1) + \mathbf{b}^{m} \\ \mathbf{h}(t) &= \alpha\, \mathbf{O}\, \mathbf{h}(t-1) + \beta\, \phi\!\left(\mathbf{W}_{\text{in}}\, \mathbf{u}(t) + \mathbf{W}_r\, \mathbf{h}(t-1) + \mathbf{W}_m\, \mathbf{m}(t-1) + \mathbf{b}\right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• mem_dims: Linear memory reservoir dimension.
• res_dims: Nonlinear reservoir hidden state dimension.
• out_dims: Output dimension.
• activation: Activation function for the nonlinear reservoir. Default: tanh.

Keyword arguments

• init_memory_reservoir: Initializer for the memory reservoir recurrent matrix C. Default: simple_cycle(; cycle_weight=1).
• init_memory_input: Initializer for the memory reservoir input matrix. Default: scaled_rand.
• init_memory_bias: Initializer for the memory reservoir bias. Default: zeros32.
• init_memory_state: Initializer used when an external memory state is not provided. Default: randn32.
• use_memory_bias: Whether the memory reservoir uses a bias term. Default: False().
• init_reservoir: Initializer for the nonlinear reservoir recurrent matrix W_r. Default: rand_sparse.
• init_input: Initializer for the nonlinear reservoir input matrix W_in. Default: scaled_rand.
• init_memory: Initializer for the matrix coupling the memory state into the nonlinear reservoir W_m. Default: scaled_rand.
• init_orthogonal: Initializer for the orthogonal skip matrix O. Default: orthogonal.
• init_bias: Initializer for the nonlinear reservoir bias, used iff use_bias=true. Default: zeros32.
• init_state: Initializer used when an external nonlinear reservoir state is not provided. Default: randn32.
• use_bias: Whether the nonlinear reservoir uses a bias term. Default: False().
• alpha: Residual skip weight α. Default: 1.0.
• beta: Nonlinear transform weight β. Default: 1.0.
• state_modifiers: A layer or collection of layers applied to the nonlinear reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state NamedTuple.

Parameters

• reservoir — parameters of the internal RMNCell, containing:
  • linear_reservoir — parameters of the memory ESNCell, including:
    • input_matrix :: (mem_dims × in_dims).
    • reservoir_matrix :: (mem_dims × mem_dims).
    • bias :: (mem_dims,) — present only if use_memory_bias=true.
  • nonlinear_reservoir — parameters of MemoryResESNCell, including:
    • input_matrix :: (res_dims × in_dims).
    • reservoir_matrix :: (res_dims × res_dims).
    • memory_matrix :: (res_dims × mem_dims).
    • orthogonal_matrix :: (res_dims × res_dims).
    • bias :: (res_dims,) — present only if use_bias=true.
• state_modifiers — a Tuple with parameters for each modifier layer; may be empty.
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims).
  • bias :: (out_dims,).

States

• reservoir — states for the internal RMNCell, containing:
  • linear_reservoir — states for the memory ESNCell.
  • nonlinear_reservoir — states for MemoryResESNCell.
  • rng — random number generator state used to sample initial recurrent states.
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

Reference

The original paper introduces this model under the name Residual Reservoir Memory Network (ResRMN); see (Ceni et al., 2025). In this package the type is named RMNResESN to keep the RMN-family naming convention consistent with RMNESN.

See also

RMNCell, MemoryResESNCell, ESNCell, LinearReadout

InputDelayESN(in_dims, res_dims, out_dims, activation=tanh;
              num_delays=1, stride=1, leak_coefficient=1.0,
              init_reservoir=rand_sparse, init_input=scaled_rand,
              init_bias=zeros32, init_state=randn32, use_bias=false,
              state_modifiers=(), readout_activation=identity,
              readout_in_dims=nothing)

Echo State Network with input delays (Fleddermann et al., 2025).

InputDelayESN composes:

1. an internal DelayLayer applied to the input signal to build tapped-delay features,
2. a stateful ESNCell (reservoir) receiving the augmented input,
3. zero or more state_modifiers applied to the reservoir state, and
4. a LinearReadout mapping the modified reservoir state to outputs.

At each time step, the input u(t) is expanded into a delay-coordinate vector that stacks the current and num_delays past inputs. This augmented signal is then used to update the reservoir state x(t).

Equations

\[\begin{aligned} \mathbf{u}_{\mathrm{d}}(t) &= \begin{bmatrix} \mathbf{u}(t) \\ \mathbf{u}(t-s) \\ \vdots \\ \mathbf{u}\!\bigl(t-Ds\bigr) \end{bmatrix}, \qquad D=\text{num\_delays},\ \ s=\text{stride}, \\ \mathbf{x}(t) &= (1-\alpha)\, \mathbf{x}(t-1) + \alpha\, \phi\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}_{\mathrm{d}}(t) + \mathbf{W}_r\, \mathbf{x}(t-1) + \mathbf{b} \right), \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right), \\ \mathbf{y}(t) &= \rho\!\left(\mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

Reservoir (passed to ESNCell):

• leak_coefficient: Leak rate in (0, 1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used iff use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.

Delay expansion (on input):

• num_delays: Number of past input states to include in the tapped-delay vector. The DelayLayer output has (num_delays + 1) * in_dims entries. Default: 1.
• stride: Delay stride in layer calls. The delay buffer is updated only when the internal clock is a multiple of stride. Default: 1.

Composition:

• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. These run after the internal DelayLayer. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• input_delay — parameters for the internal DelayLayer.
• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × ((num_delays + 1) * in_dims)) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for the user-provided modifier layers (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

States

• input_delay — state for the internal DelayLayer (its delay buffer and clock).
• reservoir — states for the internal ESNCell (e.g. rng).
• state_modifiers — states for the user-provided modifier layers.
• readout — states for LinearReadout (typically empty).

StateDelayESN(in_dims, res_dims, out_dims, activation=tanh;
         num_delays=1, stride=1, leak_coefficient=1.0,
         init_reservoir=rand_sparse, init_input=scaled_rand,
         init_bias=zeros32, init_state=randn32, use_bias=false,
         state_modifiers=(), readout_activation=identity,
         readout_in_dims=nothing)

Echo State Network with state delays (Fleddermann et al., 2025).

StateDelayESN composes:

1. a stateful ESNCell (reservoir),
2. a DelayLayer applied to the reservoir state to build tapped-delay features,
3. zero or more additional state_modifiers applied after the delay, and
4. a LinearReadout mapping delayed reservoir features to outputs.

At each time step, the reservoir produces a state vector h(t) of length res_dims. The DelayLayer then constructs a feature vector that stacks h(t) together with num_delays past states, spaced according to stride, before passing it on to any further modifiers and the readout.

Equations

\[\begin{aligned} \mathbf{x}(t) &= (1-\alpha)\, \mathbf{x}(t-1) + \alpha\, \phi\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t) + \mathbf{W}_r\, \mathbf{x}(t-1) + \mathbf{b} \right), \\ \mathbf{x}_{\mathrm{d}}(t) &= \begin{bmatrix} \mathbf{x}(t) \\ \mathbf{x}(t-s) \\ \vdots \\ \mathbf{x}\!\bigl(t-Ds\bigr) \end{bmatrix}, \qquad D=\text{num\_delays},\ \ s=\text{stride}, \\ \mathbf{z}(t) &= \psi\!\left(\mathrm{Mods}\!\left( \mathbf{x}_{\mathrm{d}}(t)\right)\right), \\ \mathbf{y}(t) &= \rho\!\left(\mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

Reservoir (passed to ESNCell):

• leak_coefficient: Leak rate in (0, 1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used iff use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.

Delay expansion:

• num_delays: Number of past reservoir states to include in the tapped-delay vector. The DelayLayer output has (num_delays + 1) * res_dims entries (current state plus num_delays past states). Default: 1.
• stride: Delay stride in layer calls. The delay buffer is updated only when the internal clock is a multiple of stride. Default: 1.

Composition:

• state_modifiers: A layer or collection of layers applied to the delayed reservoir features before the readout. These run after the internal DelayLayer. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal ESNCell, including:
  • input_matrix :: (res_dims × in_dims) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for:
  1. the internal DelayLayer, and
  2. any user-provided modifier layers (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × ((num_delays + 1) * res_dims)) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

• reservoir — states for the internal ESNCell (e.g. rng used to sample initial hidden states).
• state_modifiers — a Tuple with states for the internal DelayLayer (its delay buffer and clock) and each additional modifier layer.
• readout — states for LinearReadout (typically empty).

SVESM(in_dims, res_dims, out_dims, activation=tanh;
    leak_coefficient=1.0, init_reservoir=rand_sparse, init_input=scaled_rand,
    init_bias=zeros32, init_state=randn32, use_bias=false,
    state_modifiers=())

Support Vector Echo-State Machine (Shi and Han, 2007).

SVESM replaces the linear readout of an ESN with a SVMReadout, performing support vector regression in the high-dimensional reservoir state space (the "reservoir trick"). Training requires LIBSVM.jl to be loaded and a LIBSVM.AbstractSVR instance to be passed as the objective keyword to train.

Equations

\[\begin{aligned} \mathbf{x}(t) &= (1-\alpha)\, \mathbf{x}(t-1) + \alpha\, \phi\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t) + \mathbf{W}_r\, \mathbf{x}(t-1) + \mathbf{b} \right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \mathrm{SVR}\!\left(\mathbf{z}(t)\right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

Reservoir (passed to ESNCell):

• leak_coefficient: Leak rate α ∈ (0,1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.

Composition:

• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().

LIFESN(in_dims, res_dims, out_dims, activation=tanh;
    lookback_horizon=2, readout_activation=identity,
    state_modifiers=(), readout_in_dims=nothing, kwargs...)

Local Information Flow Echo State Network (Liu et al., 2025).

LIFESN composes:

1. a stateful LIFESNCell (a LocalInformationFlow-wrapped ESNCell),
2. zero or more state_modifiers applied to the reservoir state, and
3. a LinearReadout mapping reservoir features to outputs.

At each time step, the reservoir state is reconstructed from scratch using only the most recent lookback_horizon inputs, restricting the effective memory of the network to a local window.

Equations

\[\begin{aligned} \mathbf{x}(t) &= f\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t) + \mathbf{W}\, \mathbf{z}(t-1) \right) \\ \mathbf{z}(t-1) &= f\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t-1) + \mathbf{W}\, \mathbf{z}(t-2) \right) \\ &\ \vdots \\ \mathbf{z}(t-k+1) &= f\!\left( \mathbf{W}_{\text{in}}\, \mathbf{u}(t-k+1) + \mathbf{W}\, \mathbf{z}_0 \right) \\ \hat{\mathbf{x}}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \hat{\mathbf{x}}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ESNCell). Default: tanh.

Keyword arguments

Reservoir (passed to ESNCell via LIFESNCell):

• lookback_horizon: Number of most recent inputs used to reconstruct the recurrent state. Default: 2.
• leak_coefficient: Leak rate α ∈ (0,1]. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.

Composition:

• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

ResESN(in_dims, res_dims, out_dims, activation=tanh;
    alpha=1.0, beta=1.0, init_reservoir=rand_sparse, init_input=scaled_rand,
    init_bias=zeros32, init_state=randn32, use_bias=False(),
    state_modifiers=(), readout_activation=identity,
    init_orthogonal=orthogonal, readout_in_dims=nothing)

Residual Echo State Network (ResESN) (Ceni and Gallicchio, 2024).

Unlike ES2N, where the skip and nonlinear weights are coupled as (1-β) and β, ResESN decouples them into two independent scalars α and β.

Equations

\[\begin{aligned} \mathbf{x}(t) &= \alpha\, \mathbf{O}\, \mathbf{x}(t-1) + \beta\, \phi\!\left(\mathbf{W}_{\text{in}} \mathbf{u}(t) + \mathbf{W}_r \mathbf{x}(t-1) + \mathbf{b} \right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left( \mathbf{W}_{\text{out}}\, \mathbf{z}(t) + \mathbf{b}_{\text{out}} \right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation (for ResESNCell). Default: tanh.

Keyword arguments

• alpha: Residual skip weight α. Default: 1.0.
• beta: Nonlinear transform weight β. Default: 1.0.
• init_reservoir: Initializer for W_res. Default: rand_sparse.
• init_input: Initializer for W_in. Default: scaled_rand.
• init_orthogonal: Initializer for O. Default: orthogonal.
• init_bias: Initializer for reservoir bias (used if use_bias=true). Default: zeros32.
• init_state: Initializer used when an external state is not provided. Default: randn32.
• use_bias: Whether the reservoir uses a bias term. Default: false.
• state_modifiers: A layer or collection of layers applied to the reservoir state before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_in_dims: Readout input width for a custom dimension-changing modifier. nothing infers the width for Extend. Default: nothing.
• readout_activation: Activation for the linear readout. Default: identity.

Inputs

• x :: AbstractArray (in_dims, batch)

Returns

• Output y :: (out_dims, batch).
• Updated layer state (NamedTuple).

Parameters

• reservoir — parameters of the internal ResESNCell, including:
  • input_matrix :: (res_dims × in_dims) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_res
  • orthogonal_matrix :: (res_dims × res_dims) — O
  • bias :: (res_dims,) — present only if use_bias=true
• state_modifiers — a Tuple with parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout, typically:
  • weight :: (out_dims × res_dims) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

Exact field names for modifiers/readout follow their respective layer definitions.

States

• reservoir — states for the internal ResESNCell (e.g. rng used to sample initial hidden states).
• state_modifiers — a Tuple with states for each modifier layer.
• readout — states for LinearReadout.

ContinuousESN(in_dims, res_dims, out_dims, [activation,] tspan, args...;
    use_bias = false,
    init_reservoir = rand_sparse, init_input = scaled_rand,
    init_bias = zeros32, init_state = randn32,
    equations = __continuous_esn_rhs!,
    state_modifiers = (), readout_activation = identity,
    readout_in_dims = nothing,
    kwargs...)

Continuous-time Echo State Network ((Lukoševičius, 2012)). Composes a ContinuousESNCell, optional state_modifiers, and a LinearReadout.

Equations

\[\begin{aligned} \dot{\mathbf{x}}(t) &= -\mathbf{x}(t) + \tanh\!\left( \mathbf{W}_{\text{in}}\,\mathbf{u}(t) + \mathbf{W}_r\,\mathbf{x}(t) + \mathbf{b}\right) \\ \mathbf{z}(t) &= \mathrm{Mods}\!\left(\mathbf{x}(t)\right) \\ \mathbf{y}(t) &= \rho\!\left(\mathbf{W}_{\text{out}}\,\mathbf{z}(t) + \mathbf{b}_{\text{out}}\right) \end{aligned}\]

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir (hidden state) dimension.
• out_dims: Output dimension.
• activation: Reservoir activation. Default: tanh.
• tspan: Integration interval (t0, t1) for collectstates. Length-2, strictly increasing, finite.
• args...: Forwarded to solve positionally. The solver algorithm (Tsit5(), Euler(), …) is the first element by convention.

Keyword arguments

Reservoir (passed to ContinuousESNCell):

• use_bias: Whether the reservoir uses a bias term. Default: false.
• init_reservoir: Initialiser for W_r. Default: rand_sparse.
• init_input: Initialiser for W_in. Default: scaled_rand.
• init_bias: Initialiser for b. Default: zeros32.
• init_state: Initialiser for the initial hidden state. Default: randn32.
• equations: ODE right-hand side function (dx, x, p, t) -> nothing. Default is the continuous-time leaky-integrator ESN ODE.

Composition:

• state_modifiers: Layers applied to reservoir states before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_activation: Activation for the linear readout. Default: identity.

Solve metadata:

• kwargs...: Forwarded to solve. The keys saveat, save_everystep, and dense are reserved and rejected at construction.

Parameters

• reservoir — parameters of the internal ContinuousESNCell:
  • input_matrix :: (res_dims × in_dims) — W_in
  • reservoir_matrix :: (res_dims × res_dims) — W_r
  • bias :: (res_dims,) — present only if use_bias = true
• state_modifiers — parameters for each modifier layer (may be empty).
• readout — parameters of LinearReadout:
  • weight :: (out_dims × res_dims) — W_out
  • bias :: (out_dims,) — b_out (if the readout uses bias)

States

• reservoir — states for the internal ContinuousESNCell.
• state_modifiers — states for each modifier layer (may be empty).
• readout — states for LinearReadout.

LSM(in_dims, res_dims, out_dims, tspan, args...;
    neuron=LIFNeuron(), encoder=CurrentInjection(),
    feature_map=ExponentialSpikeFilter(),
    use_bias=false, init_reservoir=dale_sparse, init_input=scaled_rand,
    init_bias=zeros32, init_state=zeros32,
    state_modifiers=(), readout_activation=identity,
    readout_in_dims=nothing, kwargs...)

Liquid State Machine ((Maass et al., 2002)): LSMCell, optional state_modifiers, and LinearReadout.

Arguments

• in_dims: Input dimension.
• res_dims: Reservoir dimension.
• out_dims: Output dimension.
• tspan: (t0, t1) for collectstates. Length-2, strictly increasing, finite.
• args...: Positional solve arguments. Solver first by convention.

Keyword arguments

Reservoir (to LSMCell):

• neuron: Default LIFNeuron.
• encoder: Default CurrentInjection.
• feature_map: Default ExponentialSpikeFilter.
• use_bias: Default false.
• init_reservoir: Default dale_sparse.
• init_input: Default scaled_rand.
• init_bias: Default zeros32.
• init_state: Default zeros32.

Composition:

• state_modifiers: Layer, vector, or tuple. Default: ().
• readout_activation: Default: identity.

Solve:

• kwargs...: Forwarded to solve. Rejected: saveat, save_everystep, dense, callback. Use dtmax ≈ tau_ref/4.

Parameters

• reservoir — LSMCell
• state_modifiers
• readout — LinearReadout

States

• reservoir
• state_modifiers
• readout

NGRC(in_dims, out_dims; num_delays=2, stride=1,
     features=(), include_input=true, init_delay=zeros32,
     readout_activation=identity, state_modifiers=(),
     ro_dims=nothing)

Next Generation Reservoir Computing (Gauthier et al., 2021).

NGRC composes:

1. a DelayLayer applied directly to the input, producing a vector containing the current input and a fixed number of past inputs,
2. a NonlinearFeaturesLayer that applies user-provided functions to this delayed vector and concatenates the results, and
3. a LinearReadout mapping the resulting feature vector to outputs.

Arguments

• in_dims: Input dimension.
• out_dims: Output dimension.

Keyword arguments

• num_delays: Number of past input vectors to include. The internal DelayLayer outputs a vector of length (num_delays + 1) * in_dims (current input plus num_delays past inputs). Default: 2.
• stride: Delay stride in layer calls. The delay buffer is updated only when the internal clock is a multiple of stride. Default: 1.
• init_delay: Initializer (or tuple of initializers) for the delay history, passed to DelayLayer. Each initializer function is called as init(rng, in_dims, 1) to fill one delay column. Default: zeros32.
• features: A function or tuple of functions (f₁, f₂, ...) used by NonlinearFeaturesLayer. Each f is called as f(x) where x is the delayed input vector. By default it is assumed that each f returns a vector of the same length as x when ro_dims is not provided. Default: empty ().
• include_input: Whether to include the raw delayed input vector itself as the first block of the feature vector (passed to NonlinearFeaturesLayer). Default: true.
• state_modifiers: Extra layers applied after the NonlinearFeaturesLayer and before the readout. Accepts a single layer, an AbstractVector, or a Tuple. Default: empty ().
• readout_activation: Activation for the linear readout. Default: identity.
• ro_dims: Input dimension of the readout. If nothing (default), it is determined exactly by probing features and state_modifiers with a zero vector of the delayed-input length and measuring the resulting feature length; this requires that block's output length not depend on the actual input values. If probing fails (e.g. a feature function errors on an all-zero input), an ArgumentError is thrown and ro_dims must be passed explicitly.

Inputs

• x :: AbstractArray (in_dims, batch) or (in_dims,)

Returns

• Output y :: (out_dims, batch) (or (out_dims,) for vector input).
• Updated layer state (NamedTuple).

polynomial_monomials(input_vector;
    degrees = 1:2)

Generate all unordered polynomial monomials of the entries in input_vector for the given set of degrees.

For each d in degrees, this function produces all degree-d monomials of the form

• degree 1: x₁, x₂, …
• degree 2: x₁², x₁x₂, x₁x₃, x₂², …
• degree 3: x₁³, x₁²x₂, x₁x₂x₃, x₂³, …

where combinations are taken with repetition and in non-decreasing index order. This means that, for example, x₁x₂ and x₂x₁ are represented only once.

The returned vector is a flat list of all such products, in a deterministic order determined by the recursive enumeration.

Arguments

• input_vector Input vector whose entries define the variables used to build monomials.

Keyword arguments

• degrees: An iterable of positive integers specifying which monomial degrees to generate. Each degree less than 1 is skipped. Default: 1:2.

Returns

• output_monomials a vector of the same type as input_vector containing all generated monomials, concatenated across the requested degrees, in a deterministic order.

chebyshev_monomials(input_vector;
    degrees = 1:2) -> Vector

Generate all unordered Chebyshev-feature monomials of the entries in input_vector for the given set of degrees (Ratas and Pyragas, 2024).

For each d in degrees, this function evaluates T_d for every input variable and produces products over every nonempty subset containing at most d distinct variables. For example:

• degree 1: T₁(x₁), T₁(x₂), …
• degree 2: T₂(x₁), T₂(x₁)T₂(x₂), T₂(x₂), …
• degree 3: T₃(x₁), T₃(x₁)T₃(x₂), T₃(x₁)T₃(x₂)T₃(x₃), …

where T_d(·) denotes the Chebyshev polynomial of the first kind of degree d.

Variable indices within a product are strictly increasing, so a variable is never repeated. This means that T_d(x₁)T_d(x₂) is represented once, while T_d(x₁)^2 is not generated. Products are ordered deterministically by degree and then lexicographically by their variable indices.

Arguments

• input_vector::AbstractVector: Input vector whose entries define the variables to which Chebyshev polynomials are applied.

Keywords

• degrees = 1:2: An iterable of positive integers specifying which Chebyshev polynomial degrees to generate. Each degree less than 1 is skipped.

Returns

• Vector: All Chebyshev-feature products concatenated across the requested degrees in a deterministic order, with the same element type as input_vector.

resetcarry!(rng, rc::ReservoirComputer, st; init_carry=nothing)
resetcarry!(rng, rc::ReservoirComputer, ps, st; init_carry=nothing)

Reset (or set) the hidden-state carry of a model in the echo state network family.

When a function is supplied as init_carry, an existing carry provides its leading dimension; otherwise the reservoir output size is used. When init_carry=nothing, the carry is cleared and the cell's initializer is used on the next call. This does not require the cell to expose an output dimension.

Arguments

• rng: Random number generator (used if a new carry is sampled/created).
• rc: A reservoir computing network model.
• st: Current model states.
• ps: Optional model parameters. Returned unchanged.

Keyword arguments

• init_carry: Controls the initialization of the new carry.
  • nothing (default): remove/clear the carry (forces the cell to reinitialize from its own init_state on next use).
  • f: a function following standard from WeightInitializers.jl

Returns

• resetcarry!(rng, rc, st; ...) -> st′: Updated states with st′.cell.carry set to nothing or (h0,).
• resetcarry!(rng, rc, ps, st; ...) -> (ps, st′): Same as above, but also returns the unchanged ps for convenience.

RECA(in_dims, out_dims, automaton;
    input_encoding=RandomMapping(),
    generations=8, state_modifiers=(),
    readout_activation=identity, readout_in_dims=nothing)

Construct a cellular–automata reservoir model.

At each time step the input vector is randomly embedded into a Cellular Automaton (CA) lattice, the CA is evolved for generations steps, and the flattened evolution (excluding the initial row) is used as the reservoir state. A linear LinearReadout maps these features to out_dims.

Arguments

• in_dims: Number of input features (rows of training data).
• out_dims: Number of output features (rows of target data).
• automaton: A CA rule/object from CellularAutomata.jl (e.g. DCA(90), DCA(30), …).

Keyword Arguments

• input_encoding: Random embedding spec with fields permutations and expansion_size. Default is RandomMapping().
• generations: Number of CA generations to evolve per time step. Default is 8.
• state_modifiers: Optional tuple/vector of additional layers applied after the CA cell and before the readout (e.g., NLAT2(), Pad(1.0), custom transforms, etc.). Functions are wrapped automatically. Default is none.
• readout_activation: Activation applied by the readout Default is identity.

RandomMapping(permutations, expansion_size)
RandomMapping(permutations; expansion_size=40)
RandomMapping(;permutations=8, expansion_size=40)

Specify the random input embedding used by the Cellular Automata reservoir. Each time step, the input vector of length in_dims is randomly placed into a larger 1D lattice of length expansion_size, and this is repeated for permutations independent lattices (blocks). The concatenation of these blocks forms the CA initial condition of length: ca_size = expansion_size * permutations. The detail of this implementation can be found in (Nichele and Molund, 2017).

Arguments

• permutations: number of independent random maps (blocks). Larger values increase feature diversity and ca_size proportionally.
• expansion_size: width of each block (the size of a single CA lattice). Larger values increase the spatial resolution and both ca_size and states_size.

Usage

This is a configuration object; it does not perform the mapping by itself. For ordinary use, pass this configuration to RECA, which lets the cellular-automata extension construct its internal mapping tables:

using ReservoirComputing, CellularAutomata, Random

rc = RECA(in_dims = 4, out_dims = 4, DCA(90);
    input_encoding = RandomMapping(permutations = 8, expansion_size = 40),
    generations = 8)

RandomMaps

Precomputed random input-mapping data used internally by RECACell and RECA. Ordinary users should pass a RandomMapping to RECA; the cellular-automata extension constructs this value from the input dimension and number of generations.

Fields

• permutations: number of independent input maps.
• expansion_size: width of each mapped cellular-automata lattice.
• generations: number of cellular-automata generations.
• maps: integer map table with one row per permutation.
• states_size: number of features emitted by the cell.
• ca_size: length of the carried cellular-automata state.