Reinforcement Learning For Spline-Based Shape Optimization Of Flow Channels In Profile Extrusion Dies

Abstract

In this contribution, we investigate the feasibility of Reinforcement Learning (RL) as learning-based algorithm for shape optimization. Reinforcement Learning is based on a trial-and-error interaction of an agent with an environment. For each action, the agent is informed about a reward and the subsequent state of the environment, but there is no information about long-term interests as classical optimization algorithms would provide. While not necessarily superior to classical, e.g., gradient-based, optimization algorithms for one single optimization problem, Reinforcement Learning techniques are expected to perform especially well on similar optimization tasks, since the agent learns a general strategy for solving a problem instead of just concentrating on the solution of a single problem. We introduce an application case for RL-based shape optimization, in which the agent can modify the geometry of a flow channel and the resulting computational mesh through a spline-based deformation method known as Free Form Deformation by manipulating the control point coordinates of the transfor- mation spline. Every action of the agent requires computing the new state of the environment by performing a high-fidelity Finite Element Method (FEM) simulation in the modified geometry. As a first step, a modular python framework for driving the interaction be- tween the RL library stable- baselines3 [1] and the in-house FEM solver has been created and tested on a 2D example. Based on this, we will present different ways to improve the performance of the RL framework, e.g, by shaping the reward function, employing a multi-environment approach as well as comparing on- and off-policy algorithms, where the latter can be expected to be more sample-efficient, all of them paving the way towards optimizing more complex, industrial geometries. Another contribution will be the training of a custom flow feature extractor, which we expect to improve the generalization properties of the agent so that it can be reused for transfer learning tasks which becomes especially useful for addressing multi- query scenarios.