NeuralOperators

Ground TruthInferred

The demonstration shown above is the Navier-Stokes equation learned by the MarkovNeuralOperator with only one time step information. The example can be found in example/FlowOverCircle.

Installation

To install NeuralOperators.jl, use the Julia package manager:

using Pkg
Pkg.add("NeuralOperators")

Usage

Fourier Neural Operator

model = Chain(
    # lift (d + 1)-dimensional vector field to n-dimensional vector field
    # here, d == 1 and n == 64
    Dense(2, 64),
    # map each hidden representation to the next by integral kernel operator
    OperatorKernel(64=>64, (16, ), FourierTransform, gelu),
    OperatorKernel(64=>64, (16, ), FourierTransform, gelu),
    OperatorKernel(64=>64, (16, ), FourierTransform, gelu),
    OperatorKernel(64=>64, (16, ), FourierTransform),
    # project back to the scalar field of interest space
    Dense(64, 128, gelu),
    Dense(128, 1),
)

Or one can just call:

model = FourierNeuralOperator(ch = (2, 64, 64, 64, 64, 64, 128, 1),
                              modes = (16,),
                              σ = gelu)

And then train as a Flux model.

loss(𝐱, 𝐲) = l₂loss(model(𝐱), 𝐲)
opt = Flux.Optimiser(WeightDecay(1.0f-4), Flux.Adam(1.0f-3))
Flux.@epochs 50 Flux.train!(loss, params(model), data, opt)

DeepONet

# tuple of Ints for branch net architecture and then for trunk net,
# followed by activations for branch and trunk respectively
model = DeepONet((32, 64, 72), (24, 64, 72), σ, tanh)

Or specify branch and trunk as separate Chain from Flux and pass to DeepONet

branch = Chain(Dense(32, 64, σ), Dense(64, 72, σ))
trunk = Chain(Dense(24, 64, tanh), Dense(64, 72, tanh))
model = DeepONet(branch, trunk)

You can again specify loss, optimization, and training parameters just as you would for a simple neural network with Flux.

loss(xtrain, ytrain, sensor) = Flux.Losses.mse(model(xtrain, sensor), ytrain)
evalcb() = @show(loss(xval, yval, grid))

learning_rate = 0.001
opt = Adam(learning_rate)
parameters = params(model)
Flux.@epochs 400 Flux.train!(loss, parameters, [(xtrain, ytrain, grid)], opt, cb = evalcb)

A more complete example using DeepONet architecture to solve Burgers' equation can be found in the examples.

Contributing

Reproducibility

The documentation of this SciML package was built using these direct dependencies,
Status `~/work/NeuralOperators.jl/NeuralOperators.jl/docs/Project.toml`
  [e30172f5] Documenter v1.2.1
  [daee34ce] DocumenterCitations v1.3.2
 [587475ba] Flux v0.13.17
  [ea5c82af] NeuralOperators v0.4.8 `~/work/NeuralOperators.jl/NeuralOperators.jl`
Info Packages marked with  have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`
and using this machine and Julia version.
Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
  Threads: 1 on 4 virtual cores
Environment:
  JULIA_PKG_SERVER_REGISTRY_PREFERENCE = eager
A more complete overview of all dependencies and their versions is also provided.
Status `~/work/NeuralOperators.jl/NeuralOperators.jl/docs/Manifest.toml`
  [a4c015fc] ANSIColoredPrinters v0.0.1
  [621f4979] AbstractFFTs v1.5.0
  [1520ce14] AbstractTrees v0.4.4
 [79e6a3ab] Adapt v3.7.2
  [dce04be8] ArgCheck v2.3.0
  [ec485272] ArnoldiMethod v0.2.0
  [a9b6321e] Atomix v0.1.0
  [a963bdd2] AtomsBase v0.3.5
 [ab4f0b2a] BFloat16s v0.4.2
  [198e06fe] BangBang v0.3.39
  [9718e550] Baselet v0.1.1
  [2027ae74] BibInternal v0.3.5
  [13533e5b] BibParser v0.2.1
  [f1be7e48] Bibliography v0.2.20
  [d1d4a3ce] BitFlags v0.1.8
  [e1450e63] BufferedStreams v1.2.1
 [fa961155] CEnum v0.4.2
  [336ed68f] CSV v0.10.12
 [052768ef] CUDA v4.4.1
  [1af6417a] CUDA_Runtime_Discovery v0.2.2
  [082447d4] ChainRules v1.58.1
  [d360d2e6] ChainRulesCore v1.19.1
  [46823bd8] Chemfiles v0.10.41
  [944b1d66] CodecZlib v0.7.3
  [35d6a980] ColorSchemes v3.24.0
  [3da002f7] ColorTypes v0.11.4
  [c3611d14] ColorVectorSpace v0.10.0
  [5ae59095] Colors v0.12.10
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v4.12.0
  [a33af91c] CompositionsBase v0.1.2
  [f0e56b4a] ConcurrentUtilities v2.3.0
  [187b0558] ConstructionBase v1.5.4
  [6add18c4] ContextVariablesX v0.1.3
  [a8cc5b0e] Crayons v4.1.1
  [9a962f9c] DataAPI v1.15.0
  [124859b0] DataDeps v0.7.12
  [a93c6f00] DataFrames v1.6.1
  [864edb3b] DataStructures v0.18.16
  [e2d170a0] DataValueInterfaces v1.0.0
  [244e2a9f] DefineSingletons v0.1.2
  [8bb1440f] DelimitedFiles v1.9.1
  [163ba53b] DiffResults v1.1.0
  [b552c78f] DiffRules v1.15.1
  [b4f34e82] Distances v0.10.11
  [ffbed154] DocStringExtensions v0.9.3
  [e30172f5] Documenter v1.2.1
  [daee34ce] DocumenterCitations v1.3.2
  [460bff9d] ExceptionUnwrapping v0.1.10
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.8.0
  [cc61a311] FLoops v0.2.1
  [b9860ae5] FLoopsBase v0.1.1
  [5789e2e9] FileIO v1.16.2
  [48062228] FilePathsBase v0.9.21
  [1a297f60] FillArrays v1.9.3
  [53c48c17] FixedPointNumbers v0.8.4
 [587475ba] Flux v0.13.17
  [f6369f11] ForwardDiff v0.10.36
  [d9f16b24] Functors v0.4.5
 [0c68f7d7] GPUArrays v8.8.1
 [46192b85] GPUArraysCore v0.1.5
 [61eb1bfa] GPUCompiler v0.21.4
  [92fee26a] GZip v0.6.2
 [7e08b658] GeometricFlux v0.13.10
  [d7ba0133] Git v1.3.0
  [c27321d9] Glob v1.3.1
 [3ebe565e] GraphSignals v0.8.6
  [86223c79] Graphs v1.9.0
  [f67ccb44] HDF5 v0.17.1
  [cd3eb016] HTTP v1.10.1
  [b5f81e59] IOCapture v0.2.4
  [7869d1d1] IRTools v0.4.12
  [c817782e] ImageBase v0.1.7
  [a09fc81d] ImageCore v0.10.1
  [4e3cecfd] ImageShow v0.3.8
  [d25df0c9] Inflate v0.1.4
  [22cec73e] InitialValues v0.3.1
  [842dd82b] InlineStrings v1.4.0
  [7d512f48] InternedStrings v0.7.0
  [41ab1584] InvertedIndices v1.3.0
  [92d709cd] IrrationalConstants v0.2.2
  [82899510] IteratorInterfaceExtensions v1.0.0
  [033835bb] JLD2 v0.4.45
  [692b3bcd] JLLWrappers v1.5.0
  [682c06a0] JSON v0.21.4
  [0f8b85d8] JSON3 v1.14.0
  [7d188eb4] JSONSchema v1.3.0
  [b14d175d] JuliaVariables v0.2.4
  [63c18a36] KernelAbstractions v0.9.16
  [929cbde3] LLVM v6.4.2
  [b964fa9f] LaTeXStrings v1.3.1
  [0e77f7df] LazilyInitializedFields v1.2.2
  [8cdb02fc] LazyModules v0.3.1
  [2ab3a3ac] LogExpFunctions v0.3.26
  [e6f89c97] LoggingExtras v1.0.3
  [23992714] MAT v0.10.6
  [eb30cadb] MLDatasets v0.7.14
  [d8e11817] MLStyle v0.4.17
  [f1d291b0] MLUtils v0.4.4
  [3da0fdf6] MPIPreferences v0.1.10
  [1914dd2f] MacroTools v0.5.13
  [dbb5928d] MappedArrays v0.4.2
  [d0879d2d] MarkdownAST v0.1.2
  [739be429] MbedTLS v1.1.9
  [128add7d] MicroCollections v0.1.4
  [e1d29d7a] Missings v1.1.0
  [e94cdb99] MosaicViews v0.3.4
 [872c559c] NNlib v0.8.21
  [a00861dc] NNlibCUDA v0.2.7
  [15e1cf62] NPZ v0.4.3
  [77ba4419] NaNMath v1.0.2
  [71a1bf82] NameResolution v0.1.5
  [b8a86587] NearestNeighbors v0.4.16
  [ea5c82af] NeuralOperators v0.4.8 `~/work/NeuralOperators.jl/NeuralOperators.jl`
  [6fe1bfb0] OffsetArrays v1.13.0
  [0b1bfda6] OneHotArrays v0.2.4
  [4d8831e6] OpenSSL v1.4.1
 [3bd65402] Optimisers v0.2.20
  [bac558e1] OrderedCollections v1.6.3
  [5432bcbf] PaddedViews v0.5.12
  [69de0a69] Parsers v2.8.1
  [7b2266bf] PeriodicTable v1.2.0
  [fbb45041] Pickle v0.3.3
  [2dfb63ee] PooledArrays v1.4.3
  [aea7be01] PrecompileTools v1.2.0
  [21216c6a] Preferences v1.4.1
  [8162dcfd] PrettyPrint v0.2.0
  [08abe8d2] PrettyTables v2.3.1
  [33c8b6b6] ProgressLogging v0.1.4
  [74087812] Random123 v1.6.2
  [e6cf234a] RandomNumbers v1.5.3
  [c1ae055f] RealDot v0.1.0
  [189a3867] Reexport v1.2.2
  [2792f1a3] RegistryInstances v0.1.0
  [ae029012] Requires v1.3.0
  [6c6a2e73] Scratch v1.2.1
  [91c51154] SentinelArrays v1.4.1
  [efcf1570] Setfield v1.1.1
  [605ecd9f] ShowCases v0.1.0
  [777ac1f9] SimpleBufferStream v1.1.0
  [699a6c99] SimpleTraits v0.9.4
  [47aef6b3] SimpleWeightedGraphs v1.4.0
  [a2af1166] SortingAlgorithms v1.2.1
  [dc90abb0] SparseInverseSubset v0.1.2
  [276daf66] SpecialFunctions v2.3.1
  [171d559e] SplittablesBase v0.1.15
  [cae243ae] StackViews v0.1.1
  [90137ffa] StaticArrays v1.9.1
  [1e83bf80] StaticArraysCore v1.4.2
  [82ae8749] StatsAPI v1.7.0
  [2913bbd2] StatsBase v0.34.2
 [5e0ebb24] Strided v1.2.3
  [69024149] StringEncodings v0.3.7
  [892a3eda] StringManipulation v0.3.4
  [09ab397b] StructArrays v0.6.17
  [856f2bd8] StructTypes v1.10.0
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.11.1
  [62fd8b95] TensorCore v0.1.1
  [a759f4b9] TimerOutputs v0.5.23
  [3bb67fe8] TranscodingStreams v0.10.2
  [28d57a85] Transducers v0.4.80
  [bc48ee85] Tullio v0.3.7
  [9d95972d] TupleTools v1.4.3
  [5c2747f8] URIs v1.5.1
  [1986cc42] Unitful v1.19.0
  [a7773ee8] UnitfulAtomic v1.0.0
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.3
  [ea10d353] WeakRefStrings v1.4.2
  [c64b6f0f] Word2Vec v0.5.3
  [76eceee3] WorkerUtilities v1.6.1
  [ddb6d928] YAML v0.4.9
  [a5390f91] ZipFile v0.10.1
  [e88e6eb3] Zygote v0.6.69
  [700de1a5] ZygoteRules v0.2.5
 [02a925ec] cuDNN v1.1.1
 [4ee394cb] CUDA_Driver_jll v0.5.0+1
 [76a88914] CUDA_Runtime_jll v0.6.0+0
  [62b44479] CUDNN_jll v8.9.4+0
  [78a364fa] Chemfiles_jll v0.10.4+0
  [2e619515] Expat_jll v2.5.0+0
  [f5851436] FFTW_jll v3.3.10+0
  [f8c6e375] Git_jll v2.43.0+0
  [0234f1f7] HDF5_jll v1.14.3+0
  [1d5cc7b8] IntelOpenMP_jll v2024.0.2+0
  [dad2f222] LLVMExtra_jll v0.0.27+1
  [94ce4f54] Libiconv_jll v1.17.0+0
  [856f044c] MKL_jll v2024.0.0+0
  [7cb0a576] MPICH_jll v4.1.2+1
  [f1f71cc9] MPItrampoline_jll v5.3.1+1
  [9237b28f] MicrosoftMPI_jll v10.1.4+1
 [fe0851c0] OpenMPI_jll v4.1.6+0
  [458c3c95] OpenSSL_jll v3.0.12+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [9fbe4022] Word2Vec_jll v0.1.0+0
  [477f73a3] libaec_jll v1.0.6+1
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [10745b16] Statistics v1.10.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.0.5+1
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+2
  [05823500] OpenLibm_jll v0.8.1+2
  [efcefdf7] PCRE2_jll v10.42.0+1
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with  and  have new versions available. Those with  may be upgradable, but those with  are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

You can also download the manifest file and the project file.