Ordinary differential equations in machine learning : continuous mappings
Loading...
Date
2023-08-25
Authors
Γεωργαντόπουλος, Παναγιώτης
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
In this thesis the use of Ordinary Differential Equations as machine learning models such as
the Neural Ordinary Differential Equations (NODEs) and Continuous Normalizing Flows
(CNFs), is investigated. Those models constitute an alternative to neural networks based
on residual architectures whose the number of layers could be considered that result from
the sampling of the trajectory of a NODE or a CNF. It is well known that the modeling of
a highly non linear function by a deep neural network leads to a highly non linear prob-
lem that is not amenable to close form solution. By modeling a deep neural network by a
NODE, numerical integrators can be used for their training by solving in each solver’s step
the resulting adjoint equations. By doing so, in contrast to the classical back-propagation
algorithm that demands the storing of all the intermediate partial derivatives with respect
to the network’s parameters in memory, the use of constant sized memory is needed. To
showcase the performance of of NODEs and CNFs on basic machine learning problems,
we test the use of NODEs and CNFs in simple classification and non-parametric density
matching problems respectively, and compare their performances against typical ResNet
based models.
Description
Keywords
Neural ordinary differential equations, Continuous state space modeling, Continuous depth neural networks, Continuous normalizing flows