LLM-Augmented Multi-Fidelity Bayesian Optimization for Parameter Optimization in Human-Robot Collaborative Assembly

Abstract

Human-robot collaborative assembly (HRCA) is critical for manufacturing tasks like aircraft cabin drilling, where human contextual adaptability complements robotic repeatability. Parameter optimization - a critical component of HRCA systems - faces significant challenges due to limited high-fidelity experimental data, primarily resulting from costly physical trials and dynamic human-robot interaction uncertainties. While large language models (LLMs) show promise as low-cost estimators for parameter-quality relationships, their unreliable physical reasoning and hallucination-prone outputs limit direct application in safety-critical scenarios. To address these challenges, this paper proposes an LLM-augmented multi-fidelity Bayesian optimization framework for parameter optimization. This approach leverages LLMs' strengths while mitigating the scarcity of real samples and minimizing potential hallucination issues. First, a GPT-based LLM serves as the low-fidelity model, generating initial parameter-quality predictions using tailored prompts that encode domain-specific physics. Then, a latent variable Gaussian process (LVGP) hierarchically links the LLM's predictions with high-fidelity simulations through a shared covariance structure, enabling residual-based uncertainty propagation. Following this framework, an adaptive acquisition function prioritizes parameter combinations that maximize cross-fidelity consensus. Simultaneously, deviations between LLM and high-fidelity outputs trigger residual-guided prompt engineering to refine the LLM's reasoning. Failed optimization trials are systematically accumulated to iteratively retrain the LVGP kernel and adjust the LLM's prompting strategy. A case study of parameter optimization for robotic arm drilling in aircraft cabin assembly will demonstrate the advantages of the proposed method.