Many methods are available for multivariate outlier detection but until now only a hand full are developed for spatial data where there might be observations differing from their neighbors, so-called local outliers. There are methods based on a pairwise Mahalanobis distance approach, however the type of the covariance matrices used is not yet agreed upon and covers only a global covariance (Filzmoser et al. (2013)) and a very local covariance structure (Ernst and Haesbroeck (2016)), more precisely one covariance estimation per observation. To bridge the gap between the global and local approach by providing a refined covariance structure we develop spatially smoothed robust and regular covariance matrices based on the MRCD estimator (Boudt et al. (2020)) for pre-defined neighborhoods. As well known from the MCD literature, a subset of observations, the so-called H-set, is obtained by optimizing an objective function. In our case we obtain a set of optimal H-sets from minimizing an objective function which is based on a linear spatial smoothing design of local covariance matrices. A heuristic algorithm based on the notion of a C-step is developed to find the optimal set of H-sets which also shows stable convergence properties in general. We demonstrate the applicability of the new covariance estimators and the importance of a compromise between locality and globality for local outlier detection with simulated and real world data including measurements of Austrian weather stations and further compare the performance with other state-of-the-art methods from statistics and machine learning.