Self-supervised learning for robust maritime IR feature extraction

Abstract

Despite fast progress in recent years, computer vision in the marine environment stillfaces numerous challenges. Systems are susceptible to detecting large amounts of false positives, due to the small size of objects like swimmers, buoys or small waves. Due tothe motion of cameras on boats there is no guarantee for temporal consistency between detected frames. As a result of this, generalisation to out-of-distribution data also plays an important role in this domain.In this thesis, we improve the robustness of an established thermal- and vision-based object detection system by comparing and then employing self-supervised learning techniques in the domain of IR image analysis. We compare three different types of neural networkmodel architectures, i.e., ConvNeXt, Vision Transfomer and MobileNet using an adapted self-supervised learning approach, based on self-distillation that we call the Maritime Environment Self-supervised Optimization (MESO) method. Such an approach can be employed to effectively pretrain a feature extractor without the need for expensive data labeling.The key challenges of self-supervised learning in a maritime environment are: correctdata preprocessing to prevent collapse, as well as correct evaluation of the improved generalisation capabilities of the learned representations. We present different data augmentation techniques for self-supervised training on IR data, as well as different approaches for robustness evaluation. Even though a lot of effort is put into optimisations of the training method, a model that is pretrained on the popular ImageNet dataset stilloutperforms our best performing approaches in all our proposed robustness evaluation methods. Our final best performing Vision Transformer model scores 85.7% in self-supervised kNN clustering accuracy and 78.1% on linear finetuning accuracy.