neural ordinary differential equation
In plain wordsA neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.
A model obtained by taking a residual networkresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable. to the limit of infinitely many, infinitesimally thin layers. A residual update h_{t+1} = h_t + f(h_t) is one step of Euler's method for the differential equation dh/dt = f(h, t); a neural ODE keeps the equation, with a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. f_θ as its right-hand side, and computes its output by handing the equation and an initial state to a numerical solver. Training can backpropagate through the solver or use the adjoint sensitivity method, which integrates a second equation backward in time with memory independent of the number of solver steps, and adaptive solvers spend more steps where the dynamics are harder, trading accuracy against speed. Neural ODEs suit irregularly sampled time series and continuous-time dynamics and underlie continuous normalizing flowsnormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.; the framework extends to stochastic and controlled differential equations and informs flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. and diffusiondiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. samplers. Drawbacks include slow, solver-dependent training and stiff dynamics.
Because the patient records are irregularly timed, a neural ODE models the latent trajectory between visits.
Origin
Introduced by Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud of the University of Toronto in 'Neural Ordinary Differential Equations' (arXiv, June 2018; NeurIPS 2018, best-paper award). Weinan E (2017) and Eldad Haber and Lars Ruthotto (2017), among others, had interpreted residual networks as discretized dynamical systems.
Related
- residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
- normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.
- flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- time series forecastingtime series forecasting, n.: Predicting future values of something measured over time, such as sales, electricity demand, prices or rainfall, from its past values and related information.
Sources
- Chen, R. T. Q., Rubanova, Y., Bettencourt, J. & Duvenaud, D. (2018). Neural Ordinary Differential Equations. arXiv:1806.07366; NeurIPS 2018. arxiv.org/abs/1806.07366
Mentioned in
flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples., residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., liquid neural networkliquid neural network, n.: A small, brain-inspired network for time-varying data whose neurons change how quickly they react depending on their input, letting a few neurons handle tasks such as steering a car., neural operatorneural operator, n.: A neural network that learns how a whole physical system responds, such as how a fluid flows given its starting state, so it can stand in for slow simulations and answer new cases quickly., normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is., physics-informed neural networkphysics-informed neural network, n.: A neural network that is penalized during training whenever its output breaks a known physical law, such as an equation of fluid flow, so it can model a system well even from few measurements.