A Reinforcement Learning Concept for Intelligent Control in Hydropower Systems

Abstract

The expansion of volatile renewable energy sources is fundamentally reshaping the requirements placed on power system flexibility. Pumped hydro energy storage, as the most mature large-scale storage technology, is expected to play a central role in this transition — but only if its operational agility can keep pace with increasingly dynamic grid conditions. Today, the control of mode transitions in pumped storage power plants follows conservative, sequential automation with fixed safety margins, leaving significant potential for faster and more efficient operation untapped. This thesis investigates whether and how reinforcement learning (RL) can be developed and deployed for real-time process control of reversible pump turbines in pumped hydropower, and what implications this holds for operational flexibility. The work is guided by three research questions addressing the safe enablement of RL in safety-critical energy infrastructure, the identification of suitable algorithms and problem formulations, and the experimental validation of simulation-to-real transfer. The research follows a progressive trajectory across five publications. First, a conceptual framework for RL-based process control on a digital twin platform is established, combining simulation-based pre-training with transfer learning to enable safe offline development before any interaction with the physical machine. The simplified proof-of-concept is then extended to a validated multiphysics simulation model of a laboratory-scale pump turbine, implemented in MATLAB/Simulink with Simscape. Through systematic investigation ofdifferent RL algorithms, problem decompositions, and reward function designs, Proximal Policy Optimization (PPO) is identified as the most suitable algorithm, and an economically motivated cost formulation is developed that grounds the learning objective in operational reality. The culmination of this work is the first experimental demonstration of RL-based unit-level control in hydropower. A staged deployment pipeline — in which the learned policy is first distilled into a rule-based surrogate controller for safety validation before the neural network is deployed for direct online control — enables the transfer to a 400kW laboratory-scale pump turbine at TU Wien. The RL-derived control strategy reduces pump start-up time by more than 30% compared to the conventional control sequence, primarily by discovering a partially parallelized procedure that goes beyond the sequential logic of traditional automation. These results demonstrate the practical feasibility of RL-based process control in hydropower and provide a methodological blueprint applicable to other safety-critical energy systems.