Transparency techniques for neural networks trained on writer identification and writer verification

Abstract

Neural Networks are the state of the art for many tasks in the computer vision domain [SBM+17, ZTLT21]. This also includes the areas of Writer Identification and Writer Verification, where the goal is to identify the author of a handwritten text. A more novel task is the interpretability of neural networks, which are considered ”black box” systems, and to provide an explanation for the decision process of the neural network [SBM+17, HVH22, GMR+18, ZTLT21]. These explanations are used to improve the system performance, reveal possible artefacts in the training data and increase the reliability of such systems in safety-critical areas [SM19]. In this thesis, two transparencytechniques are applied to neural networks trained on Writer Identification and Writer Verification. The first transparency technique provides pixel-level saliency maps, where a significance value is assigned to each individual pixel. The second transparency technique provides two types of saliency maps, where one type shows overall similarities between two images and the second type displays similarities between one point in the first image and the overall second image. The goal is to support forensic experts with a visualization providing information on similarities in handwritten text inputs. Further, the thesis aims to explore the characteristics selected by a neural network to identify the author of a handwritten text. Three neural network architectures, namely ResNet18, ResNet20 and ResNet50, are selected based on methodologies proposed in the state of the art. The transparency techniques are adjusted for use with these specific networks and areevaluated using the deletion- and insertion score metrics. Furthermore, a qualitative evaluation is conducted, where the visualizations are compared to the areas forensic experts consider during the identification process of an author. The evaluation results show that the pixel-wise saliency map technique performs better than the point-specific saliency map technique, where the displayed highlightings are difficult to allocate to a certain character. The pixel-wise saliency maps display similar highlighting patterns for multiple occurrences of the same character, indicating a similar analysis process asapplied by a forensic expert. Overall, the pixel-wise saliency maps display characteristics suitable for the support of a forensic expert, while the point-specific saliency maps are not suitable due to non-intuitive highlightings.