Artificial Intelligence (AI) methods are widely used in the field of Earth observation and one sees an ever increasing number of publications that use deep learning or one of the many other AI methods to monitor geophysical variables from satellite measurements. As most publications predominantly highlight the positive aspects and usefulness of AI, we would like to discuss both, benefits and dangers when using AI in Earth observation. We base our work on a review of recent studies that trained AI models on satellite, model, and in situ data for either forward modelling of the satellite observations or for retrieving soil moisture. Our review shows that AI methods are very powerful, yielding accurate results for data close to their training distribution. They can beat physical and semi- empirical models that are confined by a more rigid model structure, thus being less flexible to handle unexplained behavior in the data. This makes AI methods very seductive, but the naive application of such methods can lead to unintended consequences. The most obvious case is when the data used for training the AI models are flawed, biased, or insufficient, leading to inaccurate results and wrong decision-making. Less obvious is the problem of overfitting, especially when training on high-dimensional raw data from different sources, or on data corrupted by non-stationary noise. Furthermore, labels are often at different scales, if available at all (sparse point-scale in situ observations vs. kilometer-scale model and satellite data), creating much more complex problems. Last but not least, it is notoriously difficult to understand how AI models came to their conclusions. Hence, great care must be taken in preprocessing the input data, and crafting the target loss function, to avoid noise induced local minima and unintended failure modes. In recent years, work has been done in other fields such as ML and Computer Vision to highlight, and address these problems, but the most recent developments have so far been underappreciated by the Earth observation community. In our work, we want to highlight some difficulties that arise from developing AI that is trustworthy, and applicable in the real world. For this we will show examples from using AI methods in combination with different semi-empirical models to retrieve soil moisture from Sentinel-1 and ASCAT.