Enabling Reinforcement Learning for Flexible Energy Systems Through Transfer Learning on a Digital Twin Platform

Abstract

Pumped storage power plants compensate for fluctuations in the electricity grid and improve the stability through grid services. By increasing the flexibility of pumped storage power plants, they could compensate fluctuations in an even greater extent and thus accelerate the shift to a fully renewable energy system. One way to do this is to accelerate the switching between operating modes within pumped storage stations. For this, we propose to apply reinforcement learning (RL) to control the start and stop processes within a hydraulic machine. RL has been shown to outperform traditional optimal control methods, however, safety concerns are stalling research on applying RL for process control in safety-sensitive energy systems. To enable the safe and reliable transfer of the algorithm's learning strategy from a virtual test environment to the physical asset, we present a concept for applying RL via a digital twin platform. To demonstrate this concept, we set up a simulation model for the operating behavior during the start and stop processes of a lab-scale pump-turbine and validate it with experimental data. On this virtual representation, we test the application of RL to optimally control the blow-out process within pump-turbines. We present the structure of the deep Q-learning (DQN) RL algorithm we trained and the necessary problem formulations. Our results show that the DQN algorithm is suitable for finding the optimal operating strategy to blow-out the pump-turbine runner. We discuss the viability of our approach for the control of a pump-turbine and outline the next steps to test RL on a lab-scale model machine.