Using Reinforcement Learning to Optimize Operational Strategies for Wind Energy Systems

Abstract

As the number of renewable energy systems and the energy demand are growing simultaneously, many challenges arise for wind energy systems. One challenge is to respond flexibly to market prices while the numerous actors and the gap between supply and demand are leading to increased complexity in the energy market. Also, the reliability of wind energy system components and the full utilization of any unit’s service life are gaining importance. Current research is looking for ways to balance the many factors influencing the operation of a wind turbine while maintaining the key objective of maximizing monetary profit over the life cycle. Finding the optimal operating strategy for wind farms usually requires advanced optimization methods. An emerging method that eliminates complicated mathematical optimization is reinforcement learning (RL). RL is a type of machine learning able to learn through direct interaction with the environment. RL enables finding self-learned optimal adaptive operational strategies for wind energy systems with little computational effort. To test if RL is suitable for finding optimal operation strategies for long-term value maximization in wind energy systems, we analyze the benefit of reducing turbine power through effective derating on the optimal damage budget distribution of a given failure mode over the lifetime of a wind turbine. For this, we use pre-simulated damage progression data created from aero-elastic simulations for calculating the damages within the turbine blade depending on wind speed and power output reduction. The schematic RL problem formulation is shown in Figure 2. We assess the best way to set up the RL problem by analyzing the optimal state and reward formulations and comparing the performance of different algorithms with different hyperparameter settings. We present our results for a possible lifetime extension, and thus value maximation, through efficient derating. Further, we compare our results with the optimal planning results for the respective failure modes obtained with a mathematical optimization model and discuss advantages and drawbacks of RL versus traditional optimization approaches. As the benefits of RL are most obvious as operational decision-making gets more complex, we want to focus our future research on applying RL algorithms using deep neural networks and multi-agent algorithms to consider market prices and other factors influencing the optimal operation of a wind farm.