Reinforcement learning based robot control for resource constrained microcontrollers
Loading...
Date
2026-09-23
Authors
Παπαγεωργίου, Γεώργιος
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Quadrotors are among the most widely used platforms in robotics thanks to their simplicity, agility and vertical take-off and landing capability. Nano-quadrotors such as the Bitcraze Crazyflie extend these capabilities to indoor and swarm applications, at the cost of severely limited computational resources. Controlling them is challenging: the dynamics are unstable, nonlinear and coupled, and become even more complex once motor delay and aerodynamic effects are modeled. The default cascaded PID controller of the Crazyflie is computationally cheap, but it has no knowledge of the system model and requires careful tuning. Model Predictive Control (MPC) explicitly handles the dynamics and other constraints related to the system or task, but requires solving an optimization problem at each timestep, which is often difficult to implement.This thesis proposes a low-level controller based on Reinforcement Learning (RL) that replaces the entire control loop with a single neural network, called the policy, which maps observations to motor commands. The network has only one hidden layer of 64 units (4,296 parameters), small enough to be stored in the flash memory of the Crazyflie and to be executed onboard in real time at a frequency of 250 Hz.The policy is trained entirely in simulation using the Proximal Policy Optimization (PPO) algorithm. A custom dynamics model that includes motor delay as well as aerodynamic effects was developed. To account for the differences between the system parameters used in simulation and those of the real system, Domain Randomization was employed, where a new plant is sampled at each episode during training, so the policy learns to account for a broad distribution of models rather than overfitting to the nominal one.The controller was evaluated first in the MuJoCo simulator, which differs from the one used for training, and then on a real Crazyflie 2.1, with no further fine-tuning. The evaluation covered three trajectory types: a circle, a Lissajous curve, and a helix, each flown ten times with progressively increasing speeds. The results show that: • The policy tracks the trajectories on the real drone at speeds up to 1.5 m/s for the circle and the helix, and at a peak speed of about 2 m/s for the Lissajous curve.• It outperforms the onboard PID controller on every trajectory shape; the PID’s error is roughly 2.7 to 3 times larger.• Domain Randomization consistently reduces the real-system root-mean-square error (RMSE) by 7–9 cm (34–39%) compared with a policy trained only on thenominal model.• The policy handles a light payload of up to 5 g and can also control the yaw angle of the drone.The work also delivers a custom Crazyflie client that is similar to the one offered by Bitcraze, but can also send the trajectory horizon and the desired rotation matrix.
Description
Keywords
Robotics, Machine learning, Microcontrollers