Many studies have proven the value of machine learning and remote sensing for crop yield forecasting. However, in years of severe droughts, such forecasts get unreliable. The goal of this study was to improve the reliability of the crop yield forecasts, particularly in drought years. For this purpose, the Pannonian Basin in southeastern Europe was chosen as the study area, as droughts have heavily affected agricultural production there in the last decades. Wheat and maize yields were forecasted using a random forest model on various explanatory datasets based on Earth Observation, in situ measurements, and meteorological reanalysis and forecast data. Two drought indices, Evaporative Stress Index and Standardized Precipitation-Evapotranspiration Index, were used to include information about the occurrence of droughts. The predictions were established monthly for four lead times to harvest. The first prediction was made three months before harvest and the last one at the moment of the harvest. The results were cross-validated using three-year intervals as testing sets. Years of severe droughts, 2003, 2007, 2012, and 2015, were additionally analysed. The validation showed that good predictions could be made from around two months before harvest. This was reflected in correlations of predicted and measured crop yields around 0.6. Forecasts with longer lead times than that led to significantly worse predictions. In years of severe droughts, the results were ambiguous. The wheat yield forecast model underestimates the crop losses, which led to a bad performance even shortly before harvest (correlations lower than 0.4). The maize predictions, on the other hand, showed good performances in drought years. The model underestimated crop yield losses only slightly, and the validation showed a good performance with correlations around 0.5 in the month before the harvest. Overall, a slight underestimation of drought impacts on crop losses remained for both crops at all lead times. Despite that, the results, especially for maize yields, have a considerable potential to predict crop yields reliably before the harvest and thus contribute to reducing socio-economic impacts of crop yield losses in the Pannonian Basin.