Control of uncertain input-constrained dynamical systems via reinforcement learning
Loading...
Date
2025-10-07
Authors
Τσεμπέρης, Θεόδωρος
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Control systems engineers regularly encounter nonlinearities, inevitable modeling errors and unmodeled dynamics, saturation, high dimensionality, and uncertainty displayed by dynamical systems. Control via reinforcement learning has emerged as an approach promising to confront such challenges, and is an active area of research toward the development of intelligent control systems. In this thesis, we develop such methodologies to approximately achieve optimal control of continuous-time nonlinear systems via reinforcement learning.
We begin by presenting the optimal control and dynamic programming theory, building upon calculus of variations. A glimpse of adaptive control is given to pave the way for subsequent methodologies that use similar ideas.
Then, the optimal control problem of input-affine nonlinear systems with input constraints is clearly formulated. We derive necessary and sufficient conditions for optimality based on dynamic programming. This involves solving the Hamilton-Jacobi-Bellman equation, a nonlinear partial differential equation that requires knowledge of the system dynamics and is generally intractable.
The intractability of this equation provides the impetus for searching for ways to circumvent solving it analytically or numerically. Thus, we develop tools found in the reinforcement learning literature that make the design of controllers that converge approximately to the solution of the optimal control problem in real time feasible. Essentially, the reinforcement learning algorithms that we develop are mixtures of optimal and adaptive control ideas.
Finally, we propose a novel data-driven online reinforcement learning algorithm that combines the benefits of all the previously developed controllers and converges to an approximate solution of the optimal control problem for input-affine dynamical systems with input constraints. This algorithm avoids solving the intractable Hamilton-Jacobi-Bellman equation and does not use any information about the system dynamics. We present a proof of convergence and conduct simulations that demonstrate the ability of the algorithm to correctly identify the optimal policy and value function for a nonlinear system, as well as the faster convergence achieved with orthonormal polynomials compared to the monomials commonly used in the literature.
In conclusion, this thesis aims to explore sophisticated reinforcement learning techniques to tackle the optimal control problem of nonlinear systems, and culminate in a novel algorithm that combines all the advantages of the methodologies developed by then. We hope that this will serve as inspiration for obtaining even more advanced results and achieving new breakthroughs in the future.
Description
Keywords
Optimal control, Adaptive control, Reinforcement learning