awesome-neural-ode

A collection of resources regarding the interplay between differential equations, dynamical systems, deep learning, control and scientific machine learning.

NOTE: Feel free to suggest additions via Issues or Pull Requests.

Table of Contents

• Differential Equations in Deep Learning

  • General Architectures

  • Neural ODEs

    • Training of Neural ODEs

    • Speeding up continuous models

    • Control with Neural ODEs

  • Neural SDEs

  • Neural CDEs

  • Normalizing Flows

  • Applications

• Deep Learning Methods for Differential Equations (Scientific ML)

  • Solving Differential Equations

  • Learning PDEs

  • Model Discovery

• Dynamical System View of Deep Learning

  • Recurrent Neural Networks

  • Theory and Perspectives

  • Optimization

• Software and Libraries

• Websites and Blogs

Differential Equations in Deep Learning

General Architectures

• Recurrent Neural Networks for Multivariate Time Series with Missing Values: Scientific Reports18

Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. We propose a GRU-based model called GRU-D, in which a decay mechanism is designed for the input variables and the hidden states to capture the aforementioned properties. We introduce decay rates in the model to control the decay mechanism by considering the following important factors.

• Learning unknown ODE models with Gaussian processes: arXiv18, code

However, for many complex systems it is practically impossible to determine the equations or interactions governing the underlying dynamics. In these settings, parametric ODE model cannot be formulated. Here, we overcome this issue by introducing a novel paradigm of nonparametric ODE modeling that can learn the underlying dynamics of arbitrary continuous-time systems without prior knowledge. We propose to learn non-linear, unknown differential functions from state observations using Gaussian process vector fields within the exact ODE formalism.

• Deep Equilibrium Models: NeurIPS19

We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding.

• Fast and Deep Graph Neural Networks: AAAI20

• Hamiltonian Neural Networks: NeurIPS19

• Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning: ICLR19

• Lagrangian Neural Networks: ICLR20 DeepDiffEq

• Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints: NeurIPS20, code

Reasoning about the physical world requires models that are endowed with the right inductive biases to learn the underlying dynamics. Recent works improve generalization for predicting trajectories by learning the Hamiltonian or Lagrangian of a system rather than the differential equations directly. While these methods encode the constraints of the systems using generalized coordinates, we show that embedding the system into Cartesian coordinates and enforcing the constraints explicitly with Lagrange multipliers dramatically simplifies the learning problem.

Neural ODEs

• Neural Ordinary Differential Equations (best paper award): NeurIPS18

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions

• Dissecting Neural ODEs (oral): NeurIPS20

In this work, we “open the box” and offer a system–theoretic perspective with the aim of clarifying the influence of several design choices on the underlying dynamics. We formulate and solve the infinite–dimensional problem linked to the true deep limit formulation of Neural ODE. We provide numerical approximations to the infinite–dimensional problem, leading to novel model variants, such as Galerkin and piecewise–constant Neural ODEs. Augmentation is developed beyond existing approaches to include input–layer and higher–order augmentation strategies. Finally, the novel paradigms of data–control (vector field conditioning) and depth–adaptation are introduced.

• Augmented Neural ODEs: NeurIPS19

We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.

• Graph Neural Ordinary Differential Equations: arXiv19

We introduce the framework of continuous–depth graph neural networks (GNNs). Neural graph ordinary differential equations (Neural GDEs) are formalized as the counterpart to GNNs where the input–output relationship is determined by a continuum of GNN layers, blending discrete topological structures and differential equations. We further introduce general Hybrid Neural GDE models as a hybrid dynamical systems.

• Latent ODEs for Irregularly-Sampled Time Series: NeurIPS19

• ODE2VAE: Deep generative second order ODEs with Bayesian neural networks: NeurIPS19

• Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control: arXiv19

• Stable Neural Flows: arXiv20

• On Second Order Behaviour in Augmented Neural ODEs NeurIPS20

Training of Neural ODEs

• Accelerating Neural ODEs with Spectral Elements: arXiv19

• Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE: ICML20

Speeding up continuous models

• How to Train you Neural ODE: ICML20

• Hypersolvers: Toward Fast Continuous-Depth Models: NeurIPS20

• Hey, that's not an ODE": Faster ODE Adjoints with 12 Lines of Code: arXiV20

Neural differential equations may be trained by backpropagating gradients via the adjoint method. Here, we demonstrate that the particular structure of the adjoint equations makes the usual choices of norm (such as L2) unnecessarily stringent. By replacing it with a more appropriate (semi)norm, fewer steps are unnecessarily rejected and the backpropagation is made faster.

• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs: NeurIPS20

We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method (known in the literature as “adjoint method”) to train neural ODEs on classification, density estimation, and inference approximation tasks.

Control with Neural ODEs

• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs: NeurIPS20

In this paper, we take a model-based approach to continuous-time RL, modeling the dynamics via neural ordinary differential equations (ODEs). Not only is this more sample efficient than model-free approaches, but it allows us to efficiently adapt policies learned using one schedule of interactions with the environment for another.

Neural SDEs

• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise: arXiv19

• Neural Jump Stochastic Differential Equations: arXiv19

• Towards Robust and Stable Deep Learning Algorithms for Forward Backward Stochastic Differential Equations: arXiv19

• Scalable Gradients and Variational Inference for Stochastic Differential Equations: AISTATS20

Neural CDEs

• Neural Controlled Differential Equations for Irregular Time Series (spotlight): NeurIPS20

• Neural CDEs for Long Time Series via the Log-ODE Method: arXiv20

Normalizing Flows

• Monge-Ampère Flow for Generative Modeling: arXiv18

• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models: ICLR19

• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies: arXiv18

• Flows for simultaneous manifold learning and density estimation: NeurIPS20

We introduce manifold-learning flows (M-flows), a new class of generative models that simultaneously learn the data manifold as well as a tractable probability density on that manifold. We argue why such models should not be trained by maximum likelihood alone and present a new training algorithm that separates manifold and density updates.

• TrajectoryNet: A Dynamic Optimal Transport Network for Modeling Cellular Dynamics arXiv20

• Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization: arXiv20

CP-Flows are the gradient map of a strongly convex neural potential function. The convexity implies invertibility and allows us to resort to convex optimization to solve the convex conjugate for efficient inversion.

Applications

• Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning: NeurIPS19

Deep Learning Methods for Differential Equations

Solving Differential Equations

Learning PDEs

• PDE-Net: Learning PDEs From Data: ICML18

Model Discovery

• Universal Differential Equations for Scientific Machine Learning: arXiv20

Dynamical System View of Deep Learning

Recurrent Neural Networks

• A Comprehensive Review of Stability Analysis of Continuous-Time Recurrent Neural Networks: IEEE Transactions on Neural Networks 2006

• AntysimmetricRNN: A Dynamical System View on Recurrent Neural Networks: ICLR19

• Recurrent Neural Networks in the Eye of Differential Equations: arXiv19

• Visualizing memorization in RNNs: distill19

• One step back, two steps forward: interference and learning in recurrent neural networks: arXiv18

• Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics: arXiv19

• System Identification with Time-Aware Neural Sequence Models: AAAI20

• Universality and Individuality in recurrent networks: NeurIPS19

Theory and Perspectives

• A Proposal on Machine Learning via Dynamical Systems: Communications in Mathematics and Statistics 2017

• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective: arXiv19

• Stable Architectures for Deep Neural Networks: IP17

• Beyond Finite Layer Neural Network: Bridging Deep Architects and Numerical Differential Equations: ICML18

• Review: Ordinary Differential Equations For Deep Learning: arXiv19

Optimization

• Gradient and Hamiltonian Dynamics Applied to Learning in Neural Networks: NIPS96

• Maximum Principle Based Algorithms for Deep Learning: JMLR17

• Hamiltonian Descent Methods: arXiv18

• Port-Hamiltonian Approach to Neural Network Training: CDC19, code

• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks: arXiv19

• Optimizing Millions of Hyperparameters by Implicit Differentiation: arXiv19

• Shadowing Properties of Optimization Algorithms: NeurIPS19

Software and Libraries

Python

• torchdyn: PyTorch library for all things neural differential equations. repo, docs

• torchdiffeq: Differentiable ODE solvers with full GPU support and O(1)-memory backpropagation: repo

• torchsde: Stochastic differential equation (SDE) solvers with GPU support and efficient sensitivity analysis: repo

• torchSODE: PyTorch Block-Diagonal ODE solver: repo

Julia

• DiffEqFlux: repo

Neural differential equation solvers with O(1) backprop, GPUs, and stiff+non-stiff DE solvers. Supports stiff and non-stiff neural ordinary differential equations (neural ODEs), neural stochastic differential equations (neural SDEs), neural delay differential equations (neural DDEs), neural partial differential equations (neural PDEs), and neural jump stochastic differential equations (neural jump diffusions). All of these can be solved with high order methods with adaptive time-stepping and automatic stiffness detection to switch between methods.

• NeuralNetDiffEq: Implementations of ODE, SDE, and PDE solvers via deep neural networks: repo

Websites and Blogs

• Scientific ML Blog (Chris Rackauckas and SciML): link