Optimal Energy Scheduling for Battery and Hydrogen Storage Systems Using Reinforcement Learning

Abstract

Optimal energy scheduling for sector-coupled multi-energy systems is becoming increasingly important as renewable energies such as wind and photovoltaics continue to expand. They are very volatile and difficult to predict. This creates a deviation between generation and demand that can be compensated for by energy storage technologies. For these, rule-based control is well established in industry, and mixed-integer model predictive control (MPC) is an area of research that promises the best results, usually regarding minimal costs. Drawbacks of MPC include the need for an adequate system model, often associated with high modeling effort, high computational effort for larger prediction horizons, and complications with stochastic variables. In this work, Reinforcement Learning is used in an attempt to overcome these difficulties without applying elaborate mixed-integer linear programming. The self-learning algorithm, which requires no explicit knowledge of the system behavior, can learn a control policy and uncertainties of the variables just by interaction with the (simulated) system model. It is demonstrated that Reinforcement Learning (exchange factor = 36.8 %) can learn complex system behavior with comparable quality to model predictive control (ex. = 32.4 %) and outperforms rule-based control (ex. = 41.8 %). This is done in a case study with the goal of minimizing the exchange of energy with the grid, with a battery and hydrogen system providing storage flexibility. These results were achieved in the context that the Reinforcement Learning agent only has instantaneous rather than predictive information, i.e., a very limited state of information compared to the MPC. The trained policy is then deployed while significantly decreasing the computational effort.