Towards Unified Field-Testing and Monitoring for Safe and Secure Robotic Applications

Abstract

Problems and failures that emerge in Cyber-Physical Systems (CPSs), particularly in robotic applications, may originate from various sources, including software bugs, security incidents, hardware malfunctions, and human errors. As robotic systems are deployed in various domains and application contexts, such as manufacturing sites, shop floors, agriculture, and autonomous vehicles, ensuring their safe and secure operation is a crucial aspect. While high-fidelity simulations are frequently used to validate system behavior and to perform tests, the 'simulator-to-reality gap' presents significant challenges, requiring additional field testing to validate a system under realistic conditions. As simulations alone are insufficient for performing comprehensive testing and for ensuring adherence to both functional and non-functional requirements, real-world field testing helps to alleviate these issues. However, compared to wellestablished unit testing approaches, field testing typically is still a rather ad hoc process, with insufficient support from tools and frameworks. Field tests often heavily rely on human observations, hence risking overlooking critical issues. There is a pressing need for structured, guided field-testing processes combined with adaptive runtime monitoring to capture the data required for effective error diagnosis and analysis. This paper introduces initial concepts for the Smart Unified Runtime Monitoring Infrastructure for Guided Field-Testing (SMURF) framework designed for robotic applications, combining structured test execution with automated, adaptable monitors, to ensure the efficient collection of data required for post-test analysis. Building on prior efforts in drone field-testing frameworks, we extend our scope to identify essential features for testing and monitoring ROS-based systems. Future work shall further refine this process and implement a practical framework to support developers and testers in achieving reliable, safe, and secure robotic operations.