Systematic Testing for Complex Systems in the Absence of Oracles

Abstract

Many modern software systems operate in domains where precise specifications are unavailable and reliable test oracles are difficult or impossible to obtain. This is particularly true for systems such as program analyzers, cryptographic proof systems, and machine-learning models, whose correctness depends on complex semantics or learned behavior. This keynote explores how metamorphic testing enables systematic testing in the absence of traditional oracles by checking necessary relations across multiple executions rather than individual outputs. Drawing on experience from testing program analyzers, zero-knowledge proof systems, and machine-learning models, the talk highlights the effectiveness of metamorphic testing across diverse settings. Beyond individual techniques and tools, the keynote distills general lessons on how to reason about correctness when specifications are incomplete, relational, or implicit, and how rigorous testing remains possible even when classical notions of correctness are difficult to apply.