Leveraging large language models for parametrization and code generation of impedance controllers in robotic manipulation

Abstract

Recent progress in robot learning has demonstrated that Large Language Models (LLMs) with reasoning capabilities can have high success rates in planning complex bimanual tasks.However, they take considerable time to respond, making them unusable for real-time applications. This work shows how to use their potential by building a copilot for robotics engineers to solve bimanual, contact-rich manipulation tasks in a time-efficient manner.After setting up the framework, the engineer only needs to input a natural language promptcontaining the location and characteristics of the object. The Copilot then generates a Matlab file which produces a Cartesian trajectory for each arm (zero-shot) and full parameterization of two impedance controllers, a virtual coupling spring, and end effectorrotations. The framework is implemented on two systems with different capabilities: a Franka dual arm setup and Softbanks Pepper robot. Both are successfully tested on anumber of single- and dual-arm tasks, showing the effectiveness and reliability of the framework.