Test data for dependable computer vision applications

Abstract

The goal of validation of computer vision (CV) systems is to test the reliability and robustness in various situations. This is done using dedicated test datasets which need to reflect all difficult aspects which the system will potentially face during operation. The creation of such datasets is expensive and difficult while a major challenge remains open: which aspects are actually necessary to test the robustness? This work presents a solution to plan the creation of datasets, compare the quality of existing ones, and pinpoint gaps in the data so that they may be filled using additional test cases. The CV-HAZOP checklist is the result of a risk analysis supplying over $1000$ entries which indicate potentially relevant aspects (called ``visual hazards'') for image test data. Performances of multiple stereo vision algorithms are compared between areas identified as difficult by the checklist vs. regular areas. The statistically significant drop in performance proves the value of this approach in describing challenging aspects. Existing stereo vision datasets are analysed to quantify the coverage of difficult and challenging aspects. This analysis shows: existing datasets focus on standard situations and include only a small amount of visual hazards. The visual hazard approach is then applied to the creation of a new dataset for semantic road scene understanding: Wilddash. A dedicated public benchmark webservice is created which allows the comparison of segmentation algorithm for robustness based on hazard-aware testing and negative testing. Finally, the concept is extended to panoptic segmentation and scaled to match regular state-of-the-art training datasets by creating Wilddash 2. The creation and application of a new unified label policy allows full compatibility of Wilddash 2 with three existing well-known segmentation datasets. The classifier-based detection of visual hazards allows automatic pre-selection of frames to speed up the process of creating challenging large-scale datasets.